Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought
Authors: Abulhair Saparov
He He
Venue: ICLR 2023 poster (2023)
Keywords: large language models
reasoning
question answering
chain-of-thought
in-context learning

Abstract: Large language models (LLMs) have shown remarkable reasoning capabilities given chain-of-thought prompts (examples with intermediate reasoning steps). Existing benchmarks measure reasoning ability indirectly, by evaluating accuracy on downstream tasks such as mathematical reasoning. However, it is unclear how these models obtain the answers and whether they rely on simple heuristics rather than the generated chain-of-thought. To enable systematic exploration of the reasoning ability of LLMs, we present a new synthetic question-answering dataset called PrOntoQA, where each example is generated from a synthetic world model represented in first-order logic. This allows us to parse the generated chain-of-thought into symbolic proofs for formal analysis. Our analysis on InstructGPT and GPT-3 shows that LLMs are quite capable of making correct individual deduction steps, and so are generally capable of reasoning, even in fictional contexts. However, they have difficulty with proof planning: When multiple valid deduction steps are available, they are not able to systematically explore the different options.

Published as a conference paper at ICLR 2023
LANGUAGE MODELS ARE GREEDY REASONERS: A SYSTEMATIC FORMAL ANALYSIS OF CHAIN-OF- THOUGHT
Abulhair Saparov & He He Center for Data Science, New York University, New York, NY 10011, USA {as17582, hhee &nyu.edu edu
ABSTRACT
Large language models (LLMs) have shown remarkable reasoning capabilities given chain-of-thought prompts (examples with intermediate reasoning steps). Existing benchmarks measure reasoning ability indirectly, by evaluating accuracy on downstream tasks such as mathematical reasoning. However, is unclear how these models obtain the answers and whether they rely on simple heuristics rather than the generated chain-of-thougee reasoning ability of LLMs, we present a new synthetic question-answertg called PRONTOQA, where each example generated from synthetic world model represented in first-order logic. This allows us to parse the generated chain-of- thought into symbolic proofs for formal analysis. Our analysis on INSTRUCTGPT and GPT-3 shows that LLMs are quite capable of making correct individual deduction steps, and are generally capable of reasoning, even in fictional contexts. However, they have difficulty with proof planning: When multiple valid deduction steps are available, they are not able systematically explore the different options.
To enable systematic exploration of the dataset
INTRODUCTION
The ability to reason- drawing new conclusions from provided facts is a hallmark of human intelligence. Recently, chain-of-thougll (CoT) prompting has enabled large language models (LLMs)
to perform logical reasoning tasks with impressive accuracy (Wei et al. [022 Chowdhery et al. 2022 Lewkowycz et al. 2022). In CoT prompting, each example consists of a question (c.g.. short description of the reasoning required to answer the question called the "chain-of-thoughh" (e.g.. " is 2. 2 - 1 is 1."), and a label (e.g.. "I"). When prompted with few CoT examples, the elicited reasoning allows LLMS to predict the label with much higher accuracy than standard question-answer prompting. However, it is unclear to what extent these models can reason due to several confounding factors. First, existing studies primarily rely on question-ansseeee
I?"),
(QA) tasks from real-world settings such as math word problems (Cobbe al. 2021; Han al. 2022 Weston et al. 2016). It is likely that LLMs have already acquired the knowledge through pretraining and simply retrieve the answer rather than reason over it. Second, the reasoning task may contain spurious correlations that allow the model to obtain the correct answer through shortcuts (Zhang et al., 2022b). In this work, we systematically investigate the reasoning capability of LLMs by directly evaluating their predicted chains-of-thoon th-tuhtt (the interpretable proof steps), rather than the predicted label.
To enable easy analysis of the COT, we construct new synthetic QA dataset called PRONTOQA, Duestion-Answerne
Inspired by the PROOF WRITER dataset (Tafjord et al. 202T), each example in PRONTOQA is generated from ontology and has unique proof (see figure [ for an example). We convert the proofs into syntactically simple sentences using grammar such that the inverse process is relatively easy: From the predicted COT, semantically parse each sentence into a formal language and reconstruct the underlying proof steps. We then directly analyze the model's 's reasoning by inspecting each step in the reconstructed proof and comparing them against the gold proof We emphasize here that while the dataset is an important contribution of this paper, the main contribution is the analysis that is facilitated by the dataset.
for Proof and
All analysis code, data, data generation scripts, and model outputs gi github. com/ http///apparro/oooogaaaaa
available

---

Published as a conference paper at ICLR 2023
Q: Each cat is a carnivore. Every carnivore is not kerbivorous. Carnivores are manmals. All mammals are warm-blooded. Mammals vertebrates. Every vertebrate is animal. Animals are multicellular, Fae is a cat. True or false: Fae is not herbivorous,
context
query
A: Fae is Cats are carnivores. Fae is carnivore. Every carnivore is not herbivorous, Fae is not herbivorous.
chain-of-thought label
True
FIGURE A question answering example from PRONTOQA, with each component highlighted and labeled.
Step 1:
Step 2: Generate proof from ontology
Generate ontology
Ax
Va(cat(a) + carnivore Ax
cat(fae)
animal
Hop
Ax
milticellular
carnivore(fae)
Var (a(cariivore(() + -herbivorous(3)
Hop
(berbivorous(fee)
vertebrate
Step 3:
Translate ontology to natural language context
mamnal
"Q: Each cat curnivore, Every carnivore is not herbiverous. Carnivores are mammals. All manmals are warm- bloodd. Mammals are vertebrates. Every vertebrate is an animal. Animals are multicellular"
uarm_ bloaded
carnivore
herbivorous
Step 4: Translate proof into query, chain-of- -fthoughh, and label
"Foe is cat. True or false: Fae is not herbivorous.
A: Fae is cat. Cats are carnivores. Fae is carnivore. Every carnivore not herbivorous. Fue is not herbivorous. True"
FIGURE 2: Schematic of the generative process for each example in PRONTOQA. Step 1: We generate an ontology from prior distribution, shown here tree. Each node denotes concept (e.g., mammal), each with an optional property (e.g.. warm warm_bloodee blooded), and each blue edge denotes "subtype of" relation. Step 2: Generate proof from the ontology. Each horizontal black line indicates a proof step, with its premises written above the line and the conclusion written below. Step 3: Convert the ontology into a natural language context. Step 4: Convert the proof into natural language query, chain-of-thougle and answer lahel. There is a one-to-one correspondence between the conclusion of each proof step and the sentences in the chain-of-thought
We systematically evaluate INSTRUCTGP1 Ouyang et al. 2022 and the original GPT-3 (Brown jet al., (220)) on PRONTOQA by controlling a number of variables that characterize the complexity of the reasoning task, such as the ontology type and the number of proof steps required. Our analysis shows that these models are quite good producing valid individual proof steps, even on fictional and counterfactual ontologies. However, LLMs have difficulty with proof planning: when the models encounter a point in the proof where multiple valid proof steps are available, they sometimes select the wrong step, and this often leads to an incomplete proof and subsequently an incorrect answer. Interestingly, the models are much less likely to be misled with a true ontology. suggesting that the world knowledge acquired during pretraining plays an important role in LLM reasoning. We also find that our results generalize to more sophisticated/iformaaa prompting (Wang et al. 2022). and prompts with example traces of depth- first proof search instead of COT.
prompts, such as self-consistency
2 RELATED WORK
Our proposed dataset is most closely related to PROOFWRITER Tafjord et al.) 2021 and FOLIO (Han et al. 2022) which are QA datasets designed to test reasoning ability. WRITER provides multi-hop proofs for each example. However, there are number of key properties that led to develop our own dataset (see table [ a summary). FOLIO does not provide casily-parseable Pr in their examples, and evaluation is done by inspecting the predicted labels, which may not necessarily be good measure of reasoning ability. In our analysis, we focus on more specific variables that may affect the reasoning of the model, such as: (1) Is the model's reasoning dependent on whether the example is consistent with pretraining ("true"), inconsistent ("false"), or neither ("fictional"") (2) Is the model's reasoning sensitive to whether the predicates in the examples are unary or binary? (3) Is the model's reasoning dependent on the rules of deduction in the examples? These variables are not controllable in existing datasets. Further, in some datasets, the code to generate examples is not available.
2 'NNTRRUCTGPT is the model resulting from fine-tuning GPT-3 via reinforcement learning from human feedback. Throughout the paper, "INSTRUCTGPT" refers to the model named text-davince But note that in our experiments, we also evaluate text-ada-001, bext-babbagge000 001, text-curie-00 davinci, and text-davinci--01.
2

---

Published as a conference paper at ICLR 2023
Provides easily semantically- parseable proofs
Controls for vs talse vs fictional contexts
Controls for unary vs binary predicates
Controls for specific rules of deduction
Tests reasoning beyond the domain of math word problems
Generation code available
Dataset
GSMBK
human- annotated
Cobbe
2021
X
x
X
Proo[Writer
Tatjord et al 2021
x
x
~
/
X
FDL.0
/
human- annotated
Han al al. 2022
Zhang a)
elagle
20226)
X
/
PRONTOQA
(proposed dataset)
TABLE 1: Comparison of existing datasets for the formal analysis of reasoning ability.
There are efforts to tweak or extend CoT prompting to elicit more sophisticated reasoning behavior
(Creswell et al. 2022; Wang et al. 2022 Creswell & Shanahan, 2022 Anil et al. 2022 Dohan
ct (0.2)). and they have shown that these prompting extensions CoT can improve the elicited reasoning behavior of LLMs, even with smaller models. Rather than presenting a new prompting approach, the goal of this work is to measure the reasoning ability elicited by COT. There are other datasets that have been designed or used to measure the reasoning capabilities of transformer-basee models and LLMs (Han et al. 2022 Weston al., 2016). They show that LLMs are able answer questions that require reasoning in the few-shot setting with reasonable accuracy. Similar to our approach, Betz (020)) converts logical forms into fairly simple natural language using templates. However, the examples in these datasets are consistent with the real-world, and so they may confound measuring reasoning ability with retrieval ability. Valmeekam al. (2022) found that LLMSs had difficulty with fairly simple planning task, but it is not clear whether this was due to an inability to reason or other abilities instrumental in planning, such as world modeling, keeping track of state changes, and reasoning about events that occur sequentially in time. This is despite their controlling for other variables involved in planning, such as plan generation, robustness goal formulation, among others. They experimented with examples in a "Blocksworld" environment, a significant portion of which the model can acquire from pretraining. Our work aims to address this gap. As in our approach, Dasgupta et al. (2022) specifically looked at whether LLMs can reason in fictional or counterfactual settings and found that reasoning ability is indeed negatively affected in these settings. However they did not analyze individual steps of reasoning to better understand the cause of the errors. Since we are able to formally evaluate the LLM's predicted CoT step-by-step, we are able to perform a more fine-grained analysis of their reasoning ability. Zhang et al.] a (2022b) showed that BERT is not able to learn to reason robustly, but they did not use CoT prompting and it is not obvious their results generalize to LLMs, which we evaluate.
There are two broad research approaches for reasoning in NLP: (1) reasoning over a formal symbolic language, possibly with neuro-symbolic methods and/or semantic parsing (Saparov & Mitchell 2022]
Zhang et al.) 2022a Kapanipathi et al.) 202T Dong et al. 2019 Rocktäschel & Riedel (0177). or
(2) reasoning directly over natural language (Chen et al.] 2021 Bostrom et al. 20.2, 2021110021 Welleck
al. 202T Bhagavatula
al. 2020; Angeli & Manning 2014]] MacCartney & Manning (2009).
While PRONTOQA is generated from symbolic ontologies, the examples themselves are in natural
language, and so provides value both research directions.
Recent work has examined in-context learning and found that performance on certain tasks is sensitive to the prompt (Razeghi et al. [2022 Lu et al., 2022). However, they focused on sentiment classification and simple arithmetic tasks, and it is not clear if their results generalize to reasoning. The LLM could feasibly use retrieval, rather than reasoning, to perform those tasks. Our experiments on the fictional ontology show that the model is able to reason even when there is nothing to retrieve from.
PRONTOQA: A SYNTHETIC DATASET FOR LOGICAL REASONING
We create a new dataset, called PRONTOQA for Proof and Ontology- Generated Question-Answering. where each question is generated from a symbolic ontology and proof to facilitate formal analysis of the predicted COT. To focus the scope of our exploration, and to limit the complexity of the generated questions to those within reach of current LLMs, we only consider questions that are answerable using repeated applications of the modus ponens deduction rule. More formally, modus ponens is a simple deduction rule where given the premises (z)) + g(x)) and [a) we conclude g(a) (e.g., given "All cats are carnivores" and "Fae is cat," conclude "Fae is carnivore;" see figure
3

---

Published as a conference paper at ICLR 2023
in the appendix) This rule can be easily chained together to construct proofs with controllable size. We generate CoT examples consisting of: the context, query, CoT, and label, where the context is a short paragraph containing information relevant to answer the query (see figure an example). Each example is translated from proof and ontology such that the inverse process is simple; the sentences in an example can be easily and uniquely parsed into symbolic logical forms amenable to formal analysis. More specifically, as shown in figure 2 we: (1) first generate an ontology from a set of concepts, (2) generate a proof by traversing the ontology, (3) translate the ontology into the natural language context, and (4) translate the proof into the query, CoT, and label by mapping logical forms to natural language sentences. We describe each step in further detail below.
Ontology generation. The first step is to generate small hierarchical ontology. The ontology is a set of concepts (e.g., mammal, cat, carnivore, etc) and subtype relations between them (e.g., Va(cct(2)) + carnivore())). The ontology also describes properties of concepts (e.g.. Ar (m(mmmaal( + -cold_ (rolddblooded(()) (z)oded(z))). To generate questions that are not overly complex, we restrict the ontologies to be linear (i.e., in the tree, every node has exactly 0 or 1 child nodes). Since ontologies are randomly generated, they vary in size from as few as 3 concepts to as many as 10.
Proof generation. We generate proofs from the ontology by choosing starting node uniformly at random, and generating the initial axiom indicating that an entity has a specific type (e.g.. cat(fae)). Then, we walk up the tree, with each step corresponding to an application of deduction rule (i.e., proof step). Each proof step consists of zero or more premises and one conclusion. We stop when we reach node (e.g., carnivore fae)). or a node property (e.g.. -herbivorous(fae)) such that the number of generated proof steps matches the target number of steps.
Translation to natural language example. Given a generated ontology and proof, we now translate it into natural language CoT example consisting of the question (context and query), CoT, and label. We describe how each component is generated below:
We use a simple grammar convert the formal statements of the ontology into the natural language utterances that make up the context. Every edge in the ontology is converted into sentences such as "All cats are carnivores" "Every cat carnivore." Properties of nodes are also converted into sentences of the form "All mammals are not cold-blooded," etc.
The query is generated by using the same grammar to convert the initial axiom in the proof into a natural language sentence (e.g., "Fae is a cat"). We then determine with probability 0.5 whether to ask if the conclusion of the proof is true or if its negation is false, and convert it into a natural language "true or false" query (e.g., "True or false: Fae is not herbivorous.") and label (e.g., "True").
We convert the ordered sequence of proof steps into the CoT by translating the conclusion of each proof step into a CoT sentence.
Avoiding shortcuts. In section A.2 in the appendix, we describe how we add distractor sentences in order to remove shortcuts that would allow the model to "guess" the answer without reasoning.
unique feature of PRONTOQA is that it easily programmable, with a handful of tunable knobs which we use to generate examples with varying degrees of complexity and study different aspects of reasoning in LLMs. These variables are described in greater detail in section 5.T
4 FORMAL ANALYSIS OF PREDICTED PROOFS
Instead of measuring the accuracy of the predicted answers (i.e., "true" or "false"), we would like to directly evaluate the predicted COT to check if the model derives the right answer for the right reason. We endeavor to analyze whether the model is able to apply deduction rules correctly at each proof step (i.e., local correctness), but also whether the model can plan ahead and work toward proving the answer for the query (i.e., global correctness). To measure the local correctness of a given proof step, we compute whether the step follows from one more applications of deduction rules, and whether it requires additional rules beyond those of the gold proofs. To measure the global correctness, we wish to identify proof steps that deviate from the gold proof.
To achieve this, we parse each sentence of the predicted CoT into logical form via recursive-descent parsing using the simple grammar from the generative process. We then compute whether that logical form is provable from previous logical forms via one more applications of deduction rules. This logical form corresponds to the conclusion a proof step. We then evaluate the correctness of each proof step by categorizing it according to three dimensions:
IIn natural deduction, this rule is actually composition of two steps: given (f) + >((z)), use universal climination conclude f(a) + o(a), and given fa), use implication elimination conclude g(a).
4

---

Published as a conference paper at ICLR 2023
Step type
Example (the conclusion of each step highlighted green)
Stricily-valid atomic correct step, or canonical step
"Fae is cat. Cats carnivores. Fae is a carnivore. Every carnivore not herbivorous. Fae is not herbivorous. True" (this is the gold CoTT for this example)
Strictly-valid atomic misleading step
"Fae is a cat, Cats are carnivores. Fae is a carnivore. Every carnivore is a mammal. Fae is mamunal..."
Strictly-valid non-atomic correct step
"Fue is cat. Fae is a carnivore. Every carnivore is not herbivorous. Fae is not herbivorous. True"
Strictly-valid non-atomic
"Fae is a cat. Cats are carnivores. Fae is a carnivore. Fae is a
misleading step
mammal. Every mammal is a vertebrate..."
Broadly-valis correct step
"Fae is cat. Every cat is not herbivorous. Fae is not herbivorous..."
Broadly-valid misleading
"Fae a cat. Every cat is mammal. Fue a mammal..." "
step
"Fae is cat. Cats are carnivores. Fae is a carnivore. carnivore a cat. Fae a cat..."
Every
Invalid step
TABLE 2: The types of proof steps (and examples thereof) into which we categorize each step the predicted chain-of-thought from LLMs. Compare the given chain-of-thought examples with the gold example provided in the first row.
1. Validity: Is the current proof step provable from previous steps? If is provable using only the deduction rules that appear in the gold proofs, we say the step is strictly-valid. If it is provable with a more powerful proof calculus, like natural deduction, we say the step is broadly-vate valid.. Otherwise, we say the step is invalid.
For example, given the premises, "Cats are carnivores" and "Carnivores are mammals," the step with conclusion "Cats are mammals" is broad since an additional deduction rule is required to prove it: given Vz( - g(z)) and Vz(g(x) + h(r)), conclude (z((fz) - h(z)). Notice that this is distinct from strietly-valis non-atomic step since this conclusion is not provable via
repeated applications of modus ponens. We note that this the only additional rule that we check, as we did not encounter any instances of other broadly-valió rules.
Atomicity: Is the current proof step provable from previous steps with exactly one application of a deduction rule? If so, we say the proof step is atomic. Otherwise, it is non-atomic. Note that since all broadly-valid steps are non-atomic, this distinction is only useful for strictly-valid steps. For example, given the premises. "Fae is a cat," "Cats are carnivores," and "Carnivores are mammals." the step with conclusion "Fae is a mammal" is non-atomic since the step "Fae is a carnivore" was skipped.
3. Utility: If the current proof step's 's premises are part of the gold proof, but its conclusion is not, then we say the proof step is misleading. Otherwise, it is correct.
For example, given the premises "Fac carnivore," "All carnivores are not herbivorous," and "Carnivores are mammalss"" and the goal is to prove "Fae is not herbivorous," the step "Fac is mammal" is misleading since although the step strictly-valid, it does not help to prove the goal.
The types of proof steps are listed in table 2 along with examples. Unparseable proof steps are marked as incorrect. For brevity, we refer to strictly-valid atomic correct steps as canonical steps. Psuedocode of the procedure to evaluate proofs is given in algorithm in the Appendix.
Metrics. Given the above categorization of proof steps, proof is defined to be correct if and only if there exists a path of proof steps from the premises to the conclusion (note that under this definition, it is possible for a correct proof to contain invalid proof steps). We could require that all proof steps in the path be canonical. But it is not obvious that this metric, which we call strict proof accuracy, would accurately measure the reasoning ability of the model. As such, we also consider more relaxed metrics: (a) we allow proof steps in the path to be strictly-valid non-atomic correct, which we call "skip" proof accuracy, (b) we allow proof steps to be broadly-valid valid, which we call broad proof accuracy, or (c) we allow proof steps to be strictly- or broadly-valid, which we call valid proof accuracy.
RESULTS
5.1 EXPERIMENTAL SETUP
In each experiment, we generate QA examples, perform CoT prompting on the LLMs, and analyze the predicted CoTs. We run the experiments on INSTRUCTGPT and the original GPT-3 (OpenAl
5

---

Published as a conference paper at ICLR 2023
1.0
1.0
1.0
strict proof accuracy
broad proof accuracy
"skip" proof proof accuracy
valid proof accuracy
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.6
0..
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
label accuracy
label accuracy
label accuracy
label accuracy
FIGURE 3: Scatter plots of label accuracy vs proof accuracy of all GPT-3 experiments in this paper. The black line indicates perfect agreement between label accuracy and proof accuracy. We emphasize that "proof accuracy" indicates the fraction of proofs (not proof steps) that considered correct according accuracy is not well-correlatd with strict or broad proof accuracy, and is better correlated with "skip" and valid proof accuracy. suggesting that label accuracy is a good measure of reasoning ability.
metries. Label
text corrie-0000 davinci, text- davinci-001,, text- - wwd-ddiiiii000 with greedy decoding (Ouyang et al.] 2022 Brown et al., 2020). We use 8-shot in-context learning, so each input to the LLM consists of 8 fuble-labeled questions followed by single test question with missing CoT and label. The model's s task is to predict the CoT and label for the test question. Note that all examples across all inputs are independently and identically generated from PRONTOQA.
models text- ada-001, text tesst-abbbge--00
There are a number of variables that we control when generating examples in PRONTOQA: (1) the number of hops, (2) the ordering in which the sentences are generated from the ontology, and (3) the type of the ontology. The number of hops directly controls the difficulty of the generated example, and we experiment with 1, 3. and 5 hops.
We control the ontology traversal direction: We either traverse the tree top-down (i.e., preorder) or bottom-up (i.e., postorder), generating sentence for each traversed edge/node. The ordering also affects the difficulty of the generated example: if the sentences are generated bottom-up, they will follow the same order the steps in the gold proof. On the other hand, if they are generated top-down, the order is reversed, and the task may be more difficult.
To avoid any confounding effects from knowledge acquired during pretraining, PRONTOQA generates examples with fictional concept names (e.g., "wumpus" instead of "cat," etc). But we are also interested in measuring this confounding effect, and so in addition fictional ontologies, we also generate "true" and "false" ontologies. True ontologies use real concept names and are consistent with the real-world (we randomly sample from list of three hand-coded real ontologies). False ontologies use real concept names but the trees are generated using the random process described in section 3 and it is very likely to generate false statement, such as "All mammals cats."
For each combination of variables, we run the model on 400 examples generated from the testbed, for a total of 48 experiments. We compute 95% confidence intervals for each experiment, as the number of correct proofs is distributed as Binomial (400, with p being the model's 's accuracy [Wilson 1927).
5.2 Do CORRECT ANSWERS IMPLY CORRECT REASONING?
Label accuracy may not necessarily measure whether the model is performing reasoning correctly, since the model may find ways guess the label via heuristics. To gauge whether label accuracy is a good metric and which proof accuracy metric is best to measure reasoning ability, we investigate how label accuracy is related to the various proof accuracy metrics. We plot proof accuracy vs label accuracy (i.e., simply checking whether the predicted label "True" or "False" is correct) of every experiment that we conducted in figure 3 Each point in the scatter plot corresponds to one of our 48 experiments described above. Observe that the label accuracy is poorly correlated with strict proof accuracy, and that strict proof accuracy may underestimate the model's reasoning ability. Rather, the most permissive accuracy metric has the highest correlation with label accuracy, suggesting that label accuracy is appropriate to measure reasoning accuracy. It also suggests that the most permissive proof accuracy metric most appropriate for measuring the reasoning ability of the model.
5.3 PROOF ANALYSIS RESULTS
Only the largest model is able to reason. We investigated how reasoning ability is affected by model size. In figure & in the Appendix, proof accuracy increases considerably when increasing the model size from 350M to 1.3B and 6.7B. However, only text-davinci-O00 is able perform

---

Published as a conference paper at ICLR 2023
better than chance. We were not able to conclusively discem the cause of the significant difference in performance between version 001 and 002. One possible factor is the maximum token limit of version 002 is roughly twice that of version 001. In fact, the model davinci seems to perform as well as, if not slightly better than, text- wwa--dvviiiiiooaee invalid steps decreases the model size increases, and so larger models seem to be better at making valid steps, whether or not those steps are actually useful.
In addition, we notice that the frequency of
For the remainder of the paper, our results focus sext-davinci-002 Our main results are in 4 where we show the proof accuracy and distribution of proof step types for all experiments. weal-world knowledge helps reasoning. We investigate the extent to which reasoning ability is affected by whether the ontology is fictional, "true," or "false." Evidently from figure 4 the LLM seems to perform comparably in the fictional and "false" ontology settings (accuracy is slightly worse with a "false" ontology). But when using the "true" ontology, the model performs much better, and its performance does not drop when increasing the number of hops from 3 to 5. The model is able to utilize its background knowledge from pretraining to "skip" hops, and is thus not as negatively affected by the increased hops. This is consistent with the findings of Dasgupta et al. (2022).
Evidently, the model's reasoning heavily reliant on real-world knowledge, and this may be a problem for generalizability, such as when applying LLMs to novel scenarios or to settings that are not well-represented in the training data.
Longer proofs are still challenging. We investigate the extent to which reasoning ability is affected by the number of hops in the proof. We see from figure 4 that the model handles 1- and 3-hop examples quite well but struggles with 5-hop top-down examples, with accuracy falling to chance. So while it is able to perform reasoning to extent, it is more limited the number of hops increases. Traversal direction affects reasoning. We also tested how reasoning ability is affected by the traversal direction of the ontology. We notice in figure 4 that as the number of hops increases, the model becomes sensitive to the traversal direction of the ontology (top-down vs bottom-up). This may be due to the fact that the order of the gold proof steps mirrors the bottom-up traversal, and is the reverse of the top-down traversal. Thus, the task may be made more difficult for language models if the context sentences are ordered top-down.
How do LLMs reason step-by-step? We investigate the fraction of correct and incorrect proofs that contain various types of proof steps, and whether the correctness of the proof is correlated with the presence of specific types of proof steps. Figure 4 breaks down the bars further (in darker red and blue) to indicate the fraction of proofs that contain proof steps other than canonical steps, since most predicted proof steps were canonical (in the 5-hop experiments with fictional ontology, they constitute 93.2% of proof steps). We make the following observations:
1. Most predicted proof steps are strictly-valid (in the 5-hop experiments with fictional ontology, 93.2% of proof steps are strictly-valid,, 2.4% are broadly-valid and 5.9% are invalid).
2. LLMs tend to skip steps by producing non-atomic steps. just as humans do when they verbalize their reasoning (in the 5-hop experiments with fictional ontology, 2.4% of proof steps are non- atomic, even though all steps in the few-shot examples are atomic).
Most incorrect proofs contain misleading steps and invalid steps. This suggests that the source of the incorrect reasoning is either a due to a misleading step or an invalid step that causes the model to produce steps that do not belong to the gold proof.
Intriguingly, some correct proofs also contain misleading steps and invalid steps, which implies that the model is sometimes able to recover from these "mistakes" and return to the gold proof. We
analyze this behavior in greater detail in section 5.4
5.4 WHAT LEADS TO A MISTAKE?
We investigate whether specific types of proof steps are causing INSTRUCTGPT to produce reasoning errors. To do so, we identify the first step in each incorrect proof that is not a canonical step. We observe in figure among incorrect proofs, strictly-valid atomic misleading steps appear in the proof first far more often than other non-canonical step types, including invalid steps. See figure 7 in the appendix for an example prediction where a misleading step causes the model to fail to prove the goal and produce an invalid step. This indicates that for the best-performing models, the main source of reasoning error is from misleading steps, since most predicted steps are strictly-valid and atomic. That is, imagining the space of proof steps single valid step, INSTRUCTGPT almost always performs a walk in this graph. Once INSTRUCTGPT encounters a branch where one path at the fork follows the correct proof and the other paths do not,
graph where each edge represents
7

---

Published as conference paper at ICLR 2023
strictly ty--aaldd non atonic misleading steps
smictty vall-lll atomis misd misdending steps
stristly valid non non-atome atonitiicc carrect stepe
FIGURE 4: Proof accuracy versus ontology type, number of hops, and ontology traversal direction. Each bar is subdivided into six darker bars according the types of proof steps that appear the predicted chains-of thtuugh. For example, the dark red bar corresponding to "invalid steps" indicates the proportion of incorrect proofs that contain an invalid step. The dark blue har corresponding to "invalid steps" indicates the proportion of correct proofs that contain invalid step. The proof step types detailed in figure 2
brondly validddd mistcading steps
bruadly yylulddd correct stpps
invalid stpp
Fictional ontology
1.0
incorrect
valid proof accuracy
0.8
0.6
0.4
correct
0.2
0.0
hop, bottom up traversal direction
hop, top. down traversal direction
hops, bottom up traversal direction
hops, top down traversal direction
hops, bottom up traversal direction
hops, traversal direction
False ontology
1.0
0.8
0.6
0.4
0.2
0.0
hop, bottom-up traversal direction
hop, top-down traversal direction
hops, bottom-up traversal direction
hops, top-down traversal direction
hops, bottom-up traversal direction
hops, top-down traversal direction
True ontology
1.0
0.8
0.6
0.4
0.2
0.0
hop, bottom up traversal direction
hop, top. down traversal direction
hops, bottom up traversal direction
hops, top down traversal direction
hops, bottom up traversal direction
hops, down traversal direction
INSTRUCTGPT will select the incorrect direction with some frequency and then not able to return to the correct path. Therefore. it seems that while LLMs able to produce valid proof steps with high probability, they have difficulty with proof plamning/strategizing.
We were curious if this relationship held in smaller models. We see in figure that smaller models are more prone to make invalid or non-atomic steps as their first non-canone canonical step. But as model size increases, these types of steps become rarer, and instead superseded by misleading steps.
Looking again figure 4 we note that many correct proofs also contain misleading steps, and so must be the case that INSTUCCTPPT sometimes returns to correct proof path at some point after making misleading step. To investigate this behavior more closely, we count the number of steps that model takes after making misleading step until it produces a step the gold proof and plot the histogram in figure 10 in the appendix. We observe that, in general, the more time the model spends outside the correct proof path, the less likely becomes to return the correct proof.
We demonstrate in section A.7 in the appendix that findings generalize more sophisticated prompting strategies via an experiment using celf-consisteny prompting (Wang et al.) (022) and an experiment using prompt containing example traces of depth-fire first proof search (i.e. containing examples of the search recovering from mislcading steps).
8

---

Published as conference paper at ICLR 2023
strietl- vevaald nos att-attoiic mistcading stops
atrictly -alld-aadd stomic misleading seps
striedly eeneeyy- valid noo atomitiiic currect steps
FIGURE 5: Proportion of incorrect proofs versus the type of the first error (i.e., non- canonical proof step) and model size. The proof step types detailed figure We note the 3-hop experiments
broaddly valid misteading steps
braadly valldaldd corrour steps
fictional ontology. four of 400 examples surpassed the 2049 token limit for all models (except text deee-ddvini---- 002). These examples were ignored (so the effective number of examples is 396). We omit the results for the 1-hop experiments here since there were too few incorrect proofs.
invali steps.
Fictional ontology, 3 hops, top-down traversal direction
1.00
proportion of incorrec proofs
0.75
0.25
0.00
T
text-ada-001
text-babbago-001
curie-001
text-davinci--o
False ontology, hops, top-down traversal direction
1.00
0.75
0.50
0.00
text-ada-001
text- noxt--aabbag-000
text-curie-be
davinct
text-davinct -001
text-davvinci- -002
True ontology, 3 hops, top-down traversal direction
1.00
0.50
0.25
text-ada-001
text-babbage-001
text-curie-001
davinol
text-davinci-O00
Moxt-davinc1-001
CONCLUSION AND FUTURE WORK
In this work, we introduced a synthetic fictional QA dataset called PRONTOQA designed to evaluate the reasoning ability of LLMs. We evaluated INSTRUCTGPT and GPT-3 on PRONTOQA and found that while the largest model was generally able to perform reasoning. it had difficulty with proof planning and selecting the correct proof step when there are multiple available.
PRONTOQA, and our high-level approach more broadly, could be used to compare LLM reasoning with that of humans, and to explore which aspects of human reasoning were acquired by LLMs from their pretraining. As our work has shown that LLMs are able reason to a degree, is yet unclear where the model acquired this ability. Are there portions of the pretraining data that teach the model reason? Our work shows that CoT prompting is not sufficient for more complex reasoning, such as in mathematical domains, since the reasoning tested in this work is strict subset of that of general mathematical reasoning. Mathematical proofs contain steps with much higher branching factor, where robust proof planning is instrumental. Rather, our results suggest that reasoning systems may benefit from more sophisticated proof planning/search strategies, such as neurosymbolic approaches where part of the reasoning is done over interpretable symbolic structures. PRONTOQA can be used train new reasoning systems, or pretrain/fine-tum LLMs improve their reasoning capability.
The inability of LLMs to plan ahead in their reasoning might be related recent work illuminating the theoretical computational limitations of such models (Merrill et al. 2022).
Since analysis was limited modus ponens, proof lengths of sentences, it remains to be seen whether LLMS are able to produce longer proofs, or reason with other deduction rules,
and semantically simple
semantically complex sentences/logical forms.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
All our experiments in the main text were run using the OpenAI API on September 9th, 10t and 11 2022. The self-consistency experiment was run on October 29th and 30th, and the DFS experiment was run on November 16th (see section A.7 For the sake of reproducibility of the analysis, all model outputs, the code for data generation, and the analysis code are freely available with permissive open-source license at ggithu.. com/ wwwpaaasoppooottotteaa analyze_resule py produces all figures used in this paper.
The command python
ACKNOWLEDGMENTS
We thank Vishakh Padmakumar, Richard Yuanzhe Pang, Nitish Joshi, Daniel Khashabi, Nicholas Lourie, and Will Merrill for their helpful and insightful discussion. This research was supported by Open Philanthropy, Samsung Advanced Institute of Technology (Next Generation Deep Learning: From Pattern Recognition to AD), AWS AI, and Cisco Research.
REFERENCES
Gabor Angeli and Christopher D. Manning. Naturalli: Natural logic inference for common sense reasoning. In Alessandro Moschitti, Bo Pang, and Walter Daelemans (eds.), Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, meeting of SIGDAT, a Special Interest Group of the ACL, pp. 534 545. ACL, 2014. doi: 10.3115/v1/d14-0000 1059. URL https::/t//o/// org/ 3111/v1//1111 1059.
Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay V. Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. Exploring length generalization in large language models. CoRR, abs/2207 04901, 2022. doi: 10.48550/axXi..227..00000 URL
40850/////////////// arXiv 2207 04901
Gregor Betz. Critical thinking for language models. CoRR, abs/2009.07185, 2020. URL https
//arxiv. wtgg/bs///09...... 07185
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. Abductive commonsense rea- soning. 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL [ttl:ttts///ppereeiiee http:/eee/eeeeeeeeee post/formm? By-BgglvHKDDE
Kaj Bostrom, Xinyu Zhao Swarat Chaudhuri, and Greg Durrett. Flexible generation of natural language deductions. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen- tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 6266-6266777 Association for Computational Linguistics, 2021. doi: N0.18853//1122 1/2021. emnlp-main. 506. URL https:///// org/ 10. 18663/v1/20221 emnlp www.amainn.5. 506
Kaj Bostrom, Zayne Sprague, Swarat Chaudhuri, and Greg Durrett. Natural language deduction through search over statement compositions. CoRR, abs/2201.06022 2022. URL https://
arxiv. wrg. wtgg/br//////eeeee abs//201..2....../// 06028
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voon Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL. htttps://proceeeee 4457700666fb04944411888282 142.:e44aa1444444a.aaaa wlllaa-bbbbttttttt.cca
neurips cc/ wwwppppp////a//a//////e/eee 2020/hash/
Jifan Chen, Eunsol Choi, and Greg Durrett. Can NLI models verify QA systems' predictions? In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Findings
10

---

Published as a conference paper at ICLR 2023
of the Association for Computaational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic. 16-20 November, 2021, pp. 18841-8554, Association for Computational Linguistics, 2021. doi: 00.18665/v1/222 hto.://w11222....nnnn-eee-emmmm findiing--emll.3224. URL https:///oi.org//
18653/v1/2021. findings- eertaangnseemllpppm 324
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay. Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Er- ica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language model- ing with pathways. CoRR, abs/2204.02311 2022. doi: 10445500axxi.2204..2331 URL https://doi. owg//o.//// 4.5.04850/aa5 arxiv 2204. 02311
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/21 10..44688 2021. /ttpsssst://///ttt http///axivvv/////////////.aaal/ wwwxxivvorggaaaa wttpaas....iaaaeaaa.e/aaaa.eeeea 14168
Antonia Creswell and Murray Shanahan. Faithful reasoning using large language models. CoRR, abs/2208. 14271, 2022. doi: 10.48550/a2Xi.2228..4422 URL https::////o..on org/ 485500
brXiv.2208. 14271
Antonia Creswell, Murray Shanahan, and Irina Higgins. Selection-inference: Exploiting large
language models for interpretable logical reasoning. CoRR, wwtp/225..0999111 2022. doi: 10.48550/ arXiv.2205.09711 12. URL httph:///ddo http:/...or.g//1./////////// http://xxxv2225....0.
Ishita Dasgupta, Andrew K. Lampinen, Stephanie C. Y. Chan, Antonia Creswell, Dharshan Kumaran, James L. McClelland, and Felix Hill. Language models show human-like content effects on reasoning. CoRR, abs/2207 07055, 2022. doi: 10048550aaxx..2227.0000 org/10. 48550/ 48550/arXiv. arxiv. 20002220722 07051
URLhttps:://ad
David Dohan, Winnie Xu, Aitor Lewkowycz, Jacob Austin, David Bieber, Raphael Gontijo Lopes, Yuhuai Wu, Henryk Michalewski, Rif A. Saurous, Jascha Sohl-Dickstein, Kevin Murphy, and Charles Sutton. Language model cascades. CoRR, abbp2220.,10344 2022. doi: 10.48550/arXiv. 2207.10342. URL https:/ds///o org/ 10.10000200 400.0/8550//00 arivv 2207 10342
Honghua Dong, Jiayuan Mao, Tian Lin, Chong Wang, Lihong Li, and Denny Zhou. Neural logic machines. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. http://ps///oggeereeeeeeeee/eere Bixt-hRctX
Simeng Han, Hailey Schoelkopf. Yilun Zhao, Zhenting Qi, Martin Riddell, Luke Benson, Lucy Sun, Ekaterina Zubova, Yujie Qiao, Matthew Burtell, David Peng. Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Shafiq R. Joty, Alexander R. Fabbri, Wojciech Kryscinski, Xi Victoria Lin, Caiming Xiong, and Dragomir Radev. FOLIO: natural language reasoning with first-order logic. CoRR, abs/2209. 008800, 2022. 48550/
doi: (0.48550/arxiv:2209. 00840. URL lttttt:// http:p////6ol.00/0044455/5////11 2009.00840
Pavan Kapanipathi, Ibrahim Abdelaziz, Srinivas Ravishankar, Salim Roukos, Alexander G. Gray, Ramón Fernandez Astudillo, Maria Chang, Cristina Cornelio, Saswati Dana, Achille Fokoue, Dinesh Garg, Alfio Gliozzo, Sairam Gurajada, Hima Karanam, Naweed Khan, Dinesh Khandelwal, Young-Suk Lee, Yunyao Li, Francois P. S. Luus Ndivhuwo Makondo, Nandana Mihindukula- sooriya, Tahira Naseem, Sumit Neelam, Lucian Popa, Revanth Gangi Reddy, Ryan Riegel, Gaetano Rossiello, Udit Sharma, G. P. Shrivatsa Bhargav, and Mo Yu. Leveraging abstract meaning repre- sentation for knowledge base question answering. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computational Linguistics: ACI/IJCNLP
11

---

Published as a conference paper at ICLR 2023
2021, Online Event, August 1-6, 2021, volume ACL/UCNLP 2021 of Findings of ACL. pp. 3884- 3894. Association for Computational Linguistics, 2021. doi: 10. 1/218 findiingsi-nnnnne acc.339. URL https://doi org//r/////// /t5/66///112221.ffonnnggg///////
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Sole Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. CoRR, abs/2206. 14858, 2022. doi: 10.48550/amXiv.226...88 14858. URL https //doi.org/10 48550/ arxiv. 2206. 14858
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022. Dublin, Ireland, May 22-27, 2022, pp. 0086-8098 Association for Computational Linguistics, 2022. doi:
URL https://doi. org/ 2rg/10..0 www.31861112222 acl-long
556
Bill MacCartney and Christopher D. Manning. An extended model of natural logic. In Harry Bunt, Volha Petukhova, and Sander Wubben (eds.), Proceedings of the Eight International Conference on Computational Semantics, IWCS 2009, Tilburg, The Netheriands, January 7-9, 2009, pp. 140-156. Association for Computational Linguistics, 2009. /ttpttt////acaaatooool/ org/ W09-3714/
William Merrill, Ashish Sabharwal, and Noah A. Smith. Saturated Transformers Constant-Depth Threshold Circuits. Transactions of the Association for Computational Linguistics, 10:843-856, 08 2022. ISSN 2307-387X. doi: 10. 1..116///cc___ 00493. URL https:///oo. org/o0. 101g///1//1/////// tacl a_ 00493
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models follow instructions with human feedback. CoRR. Abs/2203.001555 2022. doi: 10.488550/rx1x.2223..2222 arXiv. 2203.02255 02155
URL http::htt/s:......
Yasaman Razeghi, Robert L. Logan IV, Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies on few-shot reasoning. CoRR, abs/2202.07206, 2022. URL https:///axx org/ abs abs/2202.07206
Tim Rocktäschel and Sebastian Riedel. End-to-end differentiable proving. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach. CA, USA, pp. 3788-3800, 2017. URL https://procceddigg www.igg...uiis..... cc/ htt.//pperr/0000 2017 hass://2aa00000eaaff000501000660 Abstract. html
Abulhair Saparov and Tom M. Mitchell. Towards general natural language understanding with probabilistic worldbuilding. Trans. Assoc. Comput. Linguistics, 10:325 342, 2022. doi: 10.1162/
tacl| 00463. URL https::///aa //do:. org/100
Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. Proofwriter: Generating implications, proofs, and abductive statements over natural language. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Compulational Linguistics: ACL/IJNLPP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2021 of Findings of ACL, pp. 3621-3634, Association for Computational Linguistics, 2021. doi: 10. 10.18653/v1/202 100655///222...mdnggga1.11111 URL httts:://di. http::://do..oo/11// 10.
331.
Karthik Valmeekam, Alberto Olmo Hernandez, Sarath Sreedharan, and Subbarao Kambhampati. Large language models still can't plan (A benchmark for llms on planning and reasoning about change). CoRR, abs/2206. 10498, 2022. doi: 10,48550/arxivv22266 10498. URL/https:///a
org/10. 10. 40550/8500aa http//22.v.............2
12

---

Published as a conference paper at ICLR 2023
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V. Le, Ed H. Chi, and Denny Zhou. Self- consistency improves chain of thought reasoning in language models. CoRR, abs/2203. 2022. doi: 10.488500/r2xiv2220... 11171. URL https::/doll org/ /00g//1..m 48550/ arxiv. 2203 . 11171
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. CoRR, abs/2201.
2022. URL httpt:://aaaattt/i org/ abs /2000 11903
Sean Welleck, Jiacheng Liu, Ronan Le Bras, Hanna Hajishirzi, Yejin Choi, and Kyunghyun Cho. Naturalproofs: Mathematical theorem proving in natural language. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks I, NeurIPS Datasets and Benchmarks 2021. December 2021, proceedinge wwuripp.cc/ppppee. 2021/hash/ httpaa///cecaaaeaaaa2/aaaaaaaaa0 htw.aaaeeltt777oo--aatraaa.a.... roundd html
virtual, 2021. URL httpppps///attaatt--beenddd
round1...
Jason Weston, Antoine Bordes, Sumit Chopra, and Tomás Mikolov. Towards ai-complete question answering: A of prerequisite toy tasks. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. http::/://arae httpa/awww.orrraa///a/aaaaaaaaee maaba/1502... 05698
Edwin B Wilson. Probable inference, the law of succession, and statistical inference. J. Am. Stat. Assoc,, 22(158):209, June 1927.
Hanlin Zhang. Yi-Fan Zhang, Li Erran Li, and Eric Xing. The impact of symbolic representations on in-context learning for few-shot reasoning. Neuro Causal and Symbolic Al Workshop at NeurlPS 2022, Virtual Workshop, December 9, 2022, 2022a. URL httpps://openrevier/ net/
pdf7iddgLLdUpe(3x1 To appear.
Honghua Zhang. Liunian Harold Li, Tao Meng. Kai-Wei Chang, and Guy Van den Broeck. On the paradox of learning to reason from data. CoRR, abs/2205, 11502, 2022b. doi: 10.48550/arXiv. 2205.11502. httpps/// org 48550/ arxvv 2205. 11502
A APPENDIX
A.1 DEDUCTION RULES
Figure outlines the two deduction nules that we utilize in PRONTOQA.
Deduction rules in general form
Examples
f(a) va(f(() - g(x)) Hop
cat (fae) Vz(cat(<) carnivore(2))
Hop
g(a)
carnivore(fae)
Le., Given that "Fae is cat" and "All cats are carnivores," conclude that "Fae is carnivore."
4 Ax
Ax
cat(fae)
i.e., Assume that "Fae is cat" is axiom.
FIGURE 6: The two deduction rules that constitute the restricted proof calculus in our experiments. All proofs in PRONTOQA are composed of instances of only these two rules. Here, A is any expression, and f(() is any expression where the variable in f is substituted with any term (and similarly for g(a)).
A.2 AVOIDING SHORTCUTS
When generating examples in PRONTOQA, we have to be careful to remove any shorteuts in the question that would allow the model to "guess" the answer without reasoning. In fact, we find that without any distractors, INSTRUCTGPT is able to predict the "true"/"false" label almost perfectly. INSTRUCTGPT can utilize the heuristic that whether the queried property is mentioned the context implies whether not it is true. For instance, if the example is asking "Sally is cat. True or false: Sally is vertebrate," the model can simply look a string "Every Sneerr___i is (not) vertebrate."
13

---

Published as a conference paper at ICLR 2023
regardless of the content in the blank. Due the generative process of these examples, this kind of sentence is guaranteed to appear exactly once the context. Thus, to ensure that such heuristic not informative, we add a distractor sentence by generating novel concept that is disconnected from the ontology tree, and we assign to this new concept the negation property that queried by the question. So in the above example, if the ontology has the rule "Every mammal is vertebrate," possible distractor sentence "Every insect is not a vertebrate." " We insert this distractor sentence into a random position in the context.
A.3 EXAMPLE INSTRUCTGPT MISPREDICTION
Q: Every vumpus is a numpue. Each vumpus dull. Dumpuses are vumpuses. Every dumpus is not hot. Every impus is dumpus. Impuses are brown. Wumpuses are impuses. Wumpuses are opaque. Tumpuses are wumpuses. Tumpuses small. Every yumpus is hot. Zumpuses Rompuses are zumpuses. Every rompus is not kind. Each jompus rompus. Jompuses are sour. Alex is zumpus. True or false: Alex is hot.
tumpuses. Zumpuses
feisty,
A:
zumpus. Zumpuses are tumpuses. Alex is tumpus. Tumpuses are wumpuses. Alex is wumpus. Wumpuses are impuses. Alex an impus. Every impus is a dumpus. Alex is a dumpus. Every dumpus is not hot. Alex is not hot. False
Expected answer:
zumpus. Zumpuses are tumpuses. Alex tumpus. puses are wumpuses. Alex is wumpus. Wumpuses are impuses. Alex is an impus. Every impus is dumpus. Alex vumpus. Every vumpus is numpus. Alex is a numpus. Numpuses are jompusea. Alex jompus. Jompuses rompuses. Alex is rompus. Rompuses zumpusea. Alex is a zumpus. Zumpuses are feisty. Alex is feisty. True
Predicted answer: Alex
dumpus. Dumpuses
vumpuses. Alex is
An example from PRONTOQA hops with fictional ontology and top-down traversal direction) along with expected and predicted answer from INSTRUCTGPT. Note that most the predicted steps are canonical (highlighted yellow). The model makes a single misleading step (highlighted purple) which causes fail to prove the goal, and to eventually make invalid step (highlighted red).
FIGURE 7:
14

---

Published as a conference paper at ICLR 2023
A.4 How WE EVALUATE THE CHAIN-OF-THOUGHT
Algorithm 1: Our algorithm for reconstructing and evaluating the proof from the predicted chain-of-thought, and for computing whether each proof step valid invalid, atomic non-atomic, misleading vs correct. Here, we use the notation şfæ + c| to denote the substitution of all occurrences of the symbol æ with in the logical form ş.. We use the helper function is provable to compute whether given logical form is provable from set of axioms with one more deduction rules. The function returns tuple (P, k) where if & 2 0, is provable in k steps using the premises P. Otherwise, is not provable.
function evaluate_coo _cottee__ooootoo (context sentences Q1....,Qu
predicted chain-of-thoughe sentences C1 Ci,..., C., gold chain-of-thouge sentences . TT,) /* parse the context +/
for i E I..... do T.P = semantic.parae((()
for i E l....., do L." = semantic_parse (T;)
/* parse the gold thain-of-thought */
initialize S as empty set for i E I..... do - semantic_parse(C)
parse and evaluate the predicted chain-of thouhhhht +/
(P. k) is pr.pproablle JL,,...,IA L.,,,)
10
i 26200 0
/* if wish conditions here (e.g., requiring atomicity by checking k 1) add L S
stricter metric for proof accuracy, add
if P [ ) and : & (Lf 1
12
/+ the premises mark L a misleading step
in the gold proof but the conclusion is not */
13
14
/* the proof is correct if the final conclusion is provable */ function is_provable (logical form yp, set of axioms previous conclusions S)
return E S
15
16
if u E A
17
return (1))),
/* provable by Ax step (strictly-valid) */
else if ş0 5 return ([)).0)
/* already proved by previous step +/
20
else if o has form g(c) or -g(c) for any constants g and for E A AUS do
21
22
if has form Vo( + 7) where c)
23
(P,k) ps_provable(b/e + c), A, if 0 return (Pu (a A))
/* provable by Hop step (atrictly-valid)
26
else if ip has (a 7)
/+ note: we precompute this graph */ let G be the graph where for any axiom in A with form Vr(a - 1), and B are vertices and there is directed edge from
27
28
there a path G from di ty
/* provable with additional deduction rules (broadly-valid) +/
29
return (xxioss corresponding to path edges, length of path)
30
return (Ø,-1)
/* this step not provable (i.e., invalid) */
15

---

Published as a conference paper at ICLR 2023
strictly valid mom-atmmicc mistcaiiee mistcading steps broadly coadly-vviid valld comest steps.
stricty atomic mislcating stops
stretly wald non- con-aiimic correct sieps
A.5 PROOF ACCURACY VS MODEL SIZE
FIGURE Proof accuracy versus model size, ontology type, number of hops. Each bar is subdivided into six bars according to the types of proof steps that appear in the predicted shiins-of-thought The proof step types are detailed in figure Top-down traversal direction used in these experiments. We note that in the 3-hop experiments with fictional ontology, four the 400 examples surpassed the 2049 token limit for all models (except text-das text-davinci-002 These examples were ignored (so the effective number examples 396).
broadly braaly-ved enadly-vvlid mide midcadinng stops invlld steps
Fictional ontology, hops
correct incorrect
valid proof accuracy
1.00
0.75
0.50
0.25
0.00
text-babbage-000
davinci
daxt-davinci-022
False ontology, hops
0.75
0.50
0.00
text-ada-000
text-babbago-001
text-curie-001
davinci
text sett-davinci-011
coxt-davinci-O0
True ontology, 3 hops
1.00
0.75
0.50
0.25
0.00
toxt-ada-001
text-babbage-001
cext-curie-001
davinci
text-davinci-000
text-davinci-002
Fictional ontology, hop
1.00
0.75
0.50
0.25
0.00
mext-ada-004
Sett-babbsge-00
text-curie-001
davinci
text sest-davinci-00
davinci-002
False ontology, 1 hop
1.00
0.75
0.50
0.25
0.00
text-ada-001
sett-babbage-001
text-curie-00
davinci
cott-ddvinci-02
True ontology, hop
0.75
0.50
0.00
text-ada-001
butt-babbago-001
text-curie-001
davinci
toxt-davinci-000
text-davinci-000
16

---

Published as a conference paper at ICLR 2023
stricty vall-d non atonitoooo mistending steps
A.6 ADDITIONAL ERROR ANALYSIS
stictyy valid atomic mistending steps
stricty- enietl-valid valid non aotmitoiic correct steps
broadly ralid misecading steps
broalyy -alld currect stops
invaldd suppe
Fictional ontology
FIGURE 9: Proportion of incorrect proofs versus the type of the first erorr (i.e., non-canonical proof step), number hops, tology traversal direction. The proof step types are detailed in figure 2 We note that the 3-hop exper- iments with fictional ontol- ogy. four of the 400 exam- ples surpassed the 2049 to- ken limit for all models (ex- cept text mext-davvini--00 These examples were ig- nored (so the effective num- ber of examples 396). We omit the results for 1- hop experiments here since there were too few incorrect proofs.
proportion of incorrect prools
0.75
on-
0.25
0.00
I hops, top-down traversal direction
hops, bodumm-p traveraal direction
hops, bottomt-u traversal direction
hops, top-dlown traversal direction
False ontology
1.00
0.75
0.25
0.00
I hops, down traversal direction
hops, bottom-up up traversal direction
hops,, hotksoottompp up traversal direction
hops, down traversal direction
True ontology
1.00
0.75
0.25
ITTTI
IIIII
T
I
0.00
hops, bettom-up direction
hops, top-down
hops, bottom-up traversal direction
hops, top-down
17

---

Published as a conference paper at ICLR 2023
Fictional ontology
number of correct proofs
5
10
15
10
15
10
15
10
15
3 hops, direction
hups, top-down direction
hops. botton-up direction
hops, top-down direction
False ontology
30
FIGURE 10: Histograms de- picting the distribution the number steps each proof after a strictly-valid atomic misleading step until returning the gold proof.
11
10
15
10
15
10
15
10
15
3 hops, hottom-up direction
hops, top-down traversal direction
5 Shops, bottom-up direction
hops, top-down traversal direction
True ontology
40
30
20
10
15
10
15
10
15
10
15
3 hops, bottom-up traversal direction
hops, top-down traversal direction
5 hops, bottom-up traversal direction
hops, top-down traversal direction
Fictional ontology, hops, top-down traversal direction
40
number of correct prools
30-
20
10
5
10
15
0
5 cett--curie-00
10
15
5
10
15
15
0
5
10
15
text-ada-000 001
texts textt-bable babbbbb-bbbbbbeeeeeeeebbeeeeeeee 001
davinei
bext text-ddvvine davinci-00
text- text--avrinci davinci-002
False ontology, 3 hops, top-down traversal direction
40
30
20
10
10
15
10
15
10
15
text-adae001
taxt-babbage-001
davinct
text-davinch 1n00-00
poxt-davinnc--O0
True ontology, hops, top-down traversal direction
40
30
20
10
I
10
10
15
10
15
text--ada-001
text-babbaage-01
text-curie-00 curie-001
davinei
text-davinnii-00
twxt-davinei
18

---

Published as a conference paper at ICLR 2023
A.7 Do OTHER PROMPTING STRATEGIES HELP?
A.7.1 SELF-CONSISTENCY
To what extent do our findings generalize to prompting strategies other than CoT with greedy decoding? To test this, experimented with self-consistency prompting (Wang et al. (022), where for each example, we queried the LLM for 40 sample predictions of the CoT, using a temperature setting of 0.7. For each sample CoT 8i, we compute the following quantity:
exp
1
Σ
log Tog((81.)
16,1,... 8i5
We parse each predicted CoT sample into sequence of logical forms, and we find the logical form sequence with the highest sum of the above quantity over all CoT samples that share the same semantic parse. This logical form sequence is the final prediction.
We run this experiment our setting with hops, fictional ontology. and top-down traversal direction, with 100 examples. The resulting valid proof aceuracy is 0.56 compared to 0.545 which is not significantly different. Furthermore, inspecting specific examples of CoT samples (see figure [T]. see that for examples that the model gets wrong, the model is actually assigning higher overall probability to the incorrect proof than to the correct proof. This suggests that our results do in fact generalize to more sophisticated prompting/decoding strategies, and that strategies that endeavor to find proofs with higher probability globally (e.g. beam search) will not help the model in proof planning.
A.7.2 CAN THE MODEL LEARN TO DO DEPTH-FIRS SEARCH FROM IN-CONTEXT EXAMPLES?
Our earlier analysis also revealed possible way forward to rectify the model's 's shortcoming in proof planning: Even after making a misleading step. INSTRUCTGPT sometimes "returned" to the correct proof. We could instead relax the constraint that the touins-of-thought always reflect the shortest correct proof the answer. Instead, we allow the in-context examples to contain misleading steps, with the hope that the model learns to better recover from misleading steps. It is reasonable for humans to explore a space of possible solutions before arriving at the correct answer, and we could mimic this in LLMs by allowing the CoT to explore alternative paths in the space of proof steps, even those paths are not ultimately part of the shortest proof. This kind of search strategy is analogous to depth-first search (DFS) in graphs.
To test whether this approach improves the model's reasoning ability, we conduct an experiment where we provide in-context examples of chains-of-thee of-though that follow DFS, with the hope that the model able to learn to perform DFS when given a new test example, thereby improving the likelihood that it finds the correct answer. We run this experiment in setting with 5 hops, fictional ontology, and tow-down traversal direction, with 100 examples. The resulting valid proof accuracy is 0.55 compared to 0.545 which, again, is not significantly different.
19

---

Published as a conference paper at ICLR 2023
Q: Each rompus is wumpua. Every rompus is not opaque. Every jompus is rompus. Every jompus is not sour. Vumpuses are jompuses. Each vumpus is not earthy. Yumpuses yumpus. Tumpuses are large. Numpuses are tumpuses. Numpuses Every zumpus numpus. Dumpuses are not angry. Zumpuses are shy. Impuses are zumpuses. Impuses are luminous. Polly is an impus. True or false: Polly is angry.
vumpuses. Each yumpus is angry. Each tumpus is not cold.
Expected answer: Polly is an impus. Impuses are zumpuses. Polly is a zumpus. Every zumpus is numpus. Polly is a numpus. Numpuses are tumpuses. Polly is tumpus. Each tumpus is yumpus. Polly is yumpus. Each yumpus is angry. Polly is angry. True
Sample predicted answer: Polly is an impus. Impuses are zumpuses. Polly is a zum pus. Zumpuses numpuses. Polly numpus. Numpuses tumpuses. Polly is a tumpus. Every tumpus is yumpus. Polly is yumpus. Yumpuses are vumpuses. jompuses. Polly is jompus. Each jompus is
Polly is vumpus. Vumpuses rompus. Polly is a rompus. Rompuses are wumpuses. Polly is wumpus, Wumpuses are not angry. Polly is not angry. False Sample average log probability: -0. -0.0721235467055022
Sample predicted answer: Polly is impus. Impuses zumpuses. Polly is pus. Zumpuses are numpuses. Polly is a numpus. Numpuses are tumpuses. Polly is tumpus. Each tumpus is yumpus. Polly is yumpus. Yumpuses are angry. Polly is angry. True
Sample average log probability: -0. 003887223288008822888
Sample predicted answer: Polly is impus. Impuses zumpuses. Polly is pus. Zumpuses are numpuses. Polly is a numpus. Numpuses are tumpuses. Polly is tumpus. Each tumpus is yumpus. Polly is yumpus. Yumpuses vumpuses. Polly is a vumpus. Vumpuses are jompuses. Polly is jompus. Every jompus rompus. Polly is rompus. Rompuses are not angry Polly is not angry. False Sample average log probability: -0. 02277105944358164
FIGURE 1I: An example from PRONTOQA (5 hops with fictional ontology and top-down traversal direction) using self-consisteney (cach sample was produced using temperature 0.7), showing the expected and sample predicted answers from INSTRUCTGPT. Canonical steps are highlighted yellow, misleading steps purple, and invalid steps red. We note that the sample predicted CoTs that correspond the gold proof are given lower overall probability than those that are incorrect.
20


Reviews:
[{"review_id": "m8ER_E5WF85", "reviewer": "Reviewer_jd47", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper uses the information generated by the symbol system to help the neural system model to better reason and test CoT reasoning ability, which creates a Dataset PRONTOQA which aim at testing the reasoning ability of big model, and proposes a new method to measure the correctness of reasoning steps rather than the accuracy of reasoning answers. The experiment reveal some interesting findings of CoT. \n", "summary_of_the_review": "The process of example generation in PROTOQA is logical, systematic, interpretable and traceable. The process of logical reasoning is also very clear. This paper uses the information generated by this symbolic system to help neural system models reason better and test CoT reasoning ability. It is very innovative. Very much in line with the ICRL conference requirements. Acceptance of this paper is recommended. ", "strength_and_weaknesses": "Strengths: \n1) The generative process of example in PROTOQA is logical, systematic, interpretable and traceable. The process of logical reasoning is very clear.\n2) The interesting experiment is sufficient enough and shows us the reasoning condition of the big model.\n3) The concepts in the paper are explained clearly, and examples make it easier for readers to understand\n4) This paper allows the neural system and the symbolic system to cooperate in a better way\n\nWeaknesses: \nThe context in the dataset may be monotonous to some extent. Will the future work experiment on more diversified natural languages?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "kgyCtorH4MY", "reviewer": "Reviewer_LZJG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a PrOntoQA dataset and uses it to study the performance of chain-of-thought prompting of large language models. PrOntoQA is a synthetic reasoning dataset where each example is generated from an ontology along with the proof steps. The questions are answerable from repeated applications of the modus ponens deduction rule, which makes the reasoning process fairly simple, with the main driver of complexity coming from the number of deduction steps needed. The dataset is motivated by the desire to easily evaluate the quality of the intermediate chain-of-thought outputs, and to disentangle the reasoning performance that comes from pretraining world knowledge from the pure logical reasoning ability. The authors find that GPT-3 performs very well on individual deduction steps but can fail to output the correct chain of steps needed to solve the problem when there are multiple deduction paths possible. The authors further find that using fictional or false words instead of the true entities significantly hinders performance, thus illustrating the model's reliance on its pretraining knowledge and understanding of the entities to correctly perform the reasoning.", "summary_of_the_review": "This paper introduces a dataset that I think will be useful to the community, and the analysis illuminates some interesting observations about the reasoning abilities of LLMs. However the observations are not very novel or surprising, and the manuscript could improve in its clarity in certain parts. ", "strength_and_weaknesses": "### Strength:\n\n- The paper introduces a dataset for synthetic reasoning that can be useful for probing the reasoning behavior of LLMs. The authors take care to remove spurious heuristic solutions from the task setup, and restrict the complexity of the dataset while still maintaining a few knobs of question difficulty that one can use. \n- The paper explores several interesting notions of validity, atomicity, and misleading steps, which seems to be a fairly comprehensive perspective for studying reasoning behavior in models.\n- The paper makes interesting observations regarding the tendency of the model to skip steps and output misleading steps which are irrelevant to the final answer, and its ability to perform basic deductions well.\n\n### Weakness:\n\n- The paper makes a number of observations that have been similarly made in other papers, such as 1) the model relies on pretrained knowledge to perform reasoning (also discussed in [1] and [2]), 2) longer proofs are more challenging and correct answers can sometimes contain misleading steps (which are well known and have been noted in many papers evaluating LLMs and chain-of-thought on reasoning tasks), and 3) sentence ordering affects performance (also discussed in [3]). This limits the contributions of this work.\n\n- Several of the directions mentioned in future work seems well within the scope of the current paper and should probably be included to strengthen the current paper. For example, seeing if including prompt examples containing misleading steps would improve performance is an immediate follow-up to the observation that most errors come from misleading steps. The beam search of deduction steps is also a reasonable analysis to include in the current work in my opinion.\n\n>Our work shows that relying on CoT prompting is not sufficient for more complex reasoning, such as in mathematical domains.\n\nI don't see how this work shows the CoT prompting is insufficient for reasoning in mathematical domains.\n\n[1] https://arxiv.org/abs/2202.12837\n\n[2] https://arxiv.org/abs/2202.07206\n\n[3] https://arxiv.org/abs/2104.08786", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OvydwTeF4B", "reviewer": "Reviewer_GFj1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper releases a first order logic dataset, PRONTOQA, to evaluate the reasoning capabilities of LLMs using chain-of-thought prompting. The intermediate steps of reasoning, which are called chain-of-thought, can help in formal analysis of proofs. They also analyze results of GPT3 and  INSTRUCTGPT on this dataset.", "summary_of_the_review": "This is an interesting line of work on improving sequential reasoning. It creates a new dataset to facilitate reasoning and analysis of proofs. Although promising, it is not clear if the burden put on the user for generating prompts at every reasoning step is practical. Further, the generated proof is a validation of the system or the user input. \n\nThe discussion of the reasoning quality with respect to ontology quality is interesting. However, if ontology is available, why not use it directly as a combination of neuro-symbolic reasoning? This aspect is not explored in depth.\n\nThe evaluations are well described.", "strength_and_weaknesses": "Strengths\n● The paper is well structured and easy to read.\n● Releasing a first order logic dataset for the community to test LLMs reasoning abilities.\n\nWeakness\n● Although GPT-3 is well known in the community, InstructGPT is something not popularly known and it would have helped to have a brief description as to what tasks InstructGPT can perform, along with its pre-training data.\n● No mention of the number of entities modeled in the ontology.\n● In the results, it would have been nice to know how many samples in the test-bed are used, and how many are valid/atomic/misleading.\n● How is this work different from [1] \n● A similar work to test reasoning of LLMs is done in [2], where it was established that LLMs cannot reason, which is in contradiction with the proposal in this paper.\n\n[1]Han, Simeng, et al. \"Folio: Natural language reasoning with first-order logic.\"\narXiv preprint arXiv:2209.00840 (2022).\n\n[2]Valmeekam, Karthik, et al. \"Large Language Models Still Can't Plan (A Benchmark for\nLLMs on Planning and Reasoning about Change).\" arXiv preprint arXiv:2206.10498 (2022).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IBWWxV9U9f", "reviewer": "Reviewer_gxpW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new reasoning dataset for LLMs, and uses it to assess the logical reasoning capabilities of a number of GPT variants, using chain-of-thought prompting, while varying a number of parameters. Using this dataset, the authors show expose a number of these models’ limitations, including their poor performance on reasoning tasks requiring multiple steps.", "summary_of_the_review": "The new reasoning dataset offers some new features compared to similar existing datasets such as ProofWriter, and the paper reports a worthwhile investigation into LLM capabilities. However, the dataset is only a marginal improvement on ProofWriter IMO, and the results are unsurprising. So I don’t feel the paper offers a substantial enough contribution to merit acceptance at ICRL, given the low acceptance rate of the conference. It’s good work though, and could be a cornerstone of a larger research effort (e.g. improving the reasoning capabilities of LLMs), so the authors should be encouraged.", "strength_and_weaknesses": "The authors’ dataset is quite similar to the ProofWriter dataset released by Allen AI. But the breakdown into fictional, false and true ontologies is novel and useful, and provides insight into the models’ capabilities.\n\nI have some questions and thoughts.\n\nThe authors write: “the most permissive accuracy metric has the highest correlation with label accuracy, suggesting that GPT-3 is indeed performing the reasoning to answer the question, rather than relying on heuristics” (p.6). I don’t see how this conclusion follows. Isn’t this assuming causation from correlation? Also, since “valid proof accuracy” is strictly weaker than “strict proof accuracy”, isn’t it inevitable that the points in the scatter plot below the perfect line will be higher in the former compared to the latter? Wouldn’t this be the case however the model arrived at the answer?\n\nThe authors write: “Once InstructGPT encounters a branch where one path at the fork follows the correct proof and the other paths do not, InstructGPT will select the incorrect direction with some frequency and is then not able to return to the correct path. Therefore, it seems that while LLMs are able to produce valid proof steps with high probability, they have difficulty with proof planning/strategizing.” (pp.8-9). This account feels highly speculative. As the authors obviously know, LLMs like GPT don’t work like this at a fundamental level. They are just trying, autoregressively, to predict the next token in the given context. There is no explicit mechanism for “selecting” a branch in a proof tree, let alone for planning or strategizing, so it’s hardly surprising that they struggle with tasks where these things are required (because they are out of distribution). Maybe some cautionary remarks would be useful here.\n\nThe authors write: “we can extend the CoT paradigm where after every predicted step in the CoT, we perform beam search decoding to find the top-k most likely values for the next step. We can then perform beam search over the proof steps (i.e., sentences) rather than over the tokens.” (p.9). This has recently been done. See: A.Creswell & M.Shanahan, Faithful Reasoning Using Large Language Models, https://arxiv.org/abs/2208.14271.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-fnumUUC9m", "reviewer": "Reviewer_UrF5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose the dataset, PRONTOQA, a reasoning dataset to evaluation Language model's reasoning capabilities. The authors provide an in-depth analysis of reasoning traces.", "summary_of_the_review": "Recently there has been a lot of interested in language models (LMs) and their capabilities. Reasoning has emerged as an area of increasing interest and it's important that any capabilities are not over- (or under-) estimated. This paper offers many insights into the failings of Chain-Of-Thought prompting for LMs which I consider to be very valuable to the field. However, the (first half of the) title of the paper is misleading and the authors do not use a benchmark dataset. There are also a few points that need clarifying. \n\nI am willing to recommend acceptance of the paper, conditional on the points above being satisfactorily addressed.", "strength_and_weaknesses": "Strengths:\n1. It’s great that the authors evaluate both validity, atomicity and wether a step is misleading. These are very important metrics.\n2. Results in Figure 3 are very powerful! \n3. The three observations on page 8 are very informative.\n4. If the analysis code is made available, I can imagine this being a very useful tool for people to measure reasoning progress. However, this would be better if it had leveraged existing datasets e.g. ProofWriter.\n\nWeaknesses:\n1. The authors ignore a significant chunk of prior work. \n2. They propose a novel dataset, while there already exists a dataset, ProofWriter, that would have been suitable for these experiments and work by Betz et al. 2020 is also relevant (see detailed comments below).\n3. The title is somewhat miss-leading. I would suggest only using the second half of the title.\n4. Some of the claims are not supported, \"Reasoning is an emergent ability\".\n5. Authors only show results for a single model.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster