SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Abulhair Saparov† Richard Yuanzhe Pang† Vishakh Padmakumar† Nitish Joshi†
Seyed Mehran Kazemi
Najoung Kim $\Delta, \beta, *$
He He†,*
†New York University, ΔGoogle, βBoston University as17582@nyu.edu
Given the intractably large size of the space of proofs, any model that is capable of general deductive reasoning must generalize to proofs of greater complexity. Recent studies have shown that large language models (LLMs) possess some abstract deductive reasoning ability given chain-of-thought prompts. However, they have primarily been tested on proofs using modus ponens or of a specific size, and from the same distribution as the in-context examples. To measure the general deductive reasoning ability of LLMs, we test on a broad set of deduction rules and measure their ability to generalize to more complex proofs from simpler demonstrations from multiple angles: depth-, width-, and compositional generalization. To facilitate systematic exploration, we construct a new synthetic and programmable reasoning dataset that enables control over deduction rules and proof complexity. Our experiments on four LLMs of various sizes and training objectives show that they are able to generalize to compositional proofs. However, they have difficulty generalizing to longer proofs, and they require explicit demonstrations to produce hypothetical subproofs, specifically in proof by cases and proof by contradiction.
In many tasks that require deductive reasoning, such as theorem proving or medical diagnosis, the complexity of proofs can grow without bound via the use of multiple deduction rules and the composition of subproofs. Given the large space of proofs, it is infeasible to find data to cover proofs of all sizes. Therefore, a general reasoning model must extrapolate to complex proofs from simpler ones. Recent work has shown that LLMs, combined with in-context learning (ICL) and chain-of-thought (CoT) prompting, are capable of deductive reasoning to an extent [Huang and Chang, 2022, Han et al., 2022, Wei et al., 2022b, Kojima et al., 2022, Lewkowycz et al., 2022, Nye et al., 2021, Gontier et al., 2020]. However, much of the prior work focused on a limited set of deduction rules such as modus ponens [Zhang et al., 2022a, Saparov and He, 2023, Tafjord et al., 2021]. In addition, the evaluation is in-demonstration, where the test example comes from the same distribution as the in-context demonstrations. In this work, we evaluate whether LLMs are capable of general deductive reasoning by measuring how well they generalize to proofs that are more complex
$\dagger , \Delta$  GPT and PaLM experiments in this paper were conducted independently. Authors affiliated with NYU $^{\dagger}$  were responsible for the GPT experiments, and authors affiliated with Google $^{\Delta}$  were responsible for the PaLM experiments.
FIGURE 1: An overview of the kinds of OOD generalization that we test in our experiments. Each training example is a sample CoT demonstration provided to the LLM in the few-shot prompt, whereas each test example is a sample proof that the model is expected to output.
than their demonstrations.1
We characterize the complexity of proofs from three angles: the deduction rules involved, the depth of the proof (i.e. length of a sequential chain of proof steps), and the width of the proof (i.e. the number of premises of each proof step). Each of the three dimensions contributes to the overall size of the proof. To measure the general deductive reasoning ability of LLMs, we extend prior studies in two key ways. First, we determine whether LLMs have learned a complete set of deduction rules, beyond modus ponens. Second, we evaluate whether they can reason over longer proofs than those given as in-context examples (depth- and width- generalization); and whether they are able to use multiple different deduction rules in a single proof (compositional generalization). Figure 1 shows an overview of our study.
Our findings suggest that in-context learning is best applied to reasoning tasks by including examples that cover a diverse set of deduction rules, and keeping the examples simple. The in-context examples should especially contain examples of deduction rules that are less familiar to the model (i.e. proof by cases and proof by contradiction), and distractors should be provided for such examples as the model is more prone to overfitting.
We test four different LLMs of different scales and training objectives: GPT-3.5 175B [Ouyang et al., 2022], PaLM 540B [Chowdhery et al., 2022], LLaMA 65B [Touvron et al., 2023], and FLAN-T5 11B [Chung et al., 2022], and we find:
1. CoT is able to elicit out-of-demonstration (OOD) reasoning in LLMs generalizing to compositional proofs. This is somewhat surprising given the amount of previous work that claim that LLMs are not able to generalize compositionally [Hosseini et al., 2022, An et al., 2023]. See Section 4.2.2 and Figure 6.
2. ICL generalizes differently compared to supervised learning (i.e. gradient descent on in-context examples). We find numerous examples where it is strictly worse to provide in-context examples from the same distribution as the test example. For instance, in some cases, we observe better generalization to compositional proofs when the in-context examples each contain individual deduction rules. See Sections 4.2.2 and 4.3, and Figures 6 and 8.
3. However, the LLMs cannot generalize to some deduction rules without explicit demonstrations, specifically, proof by cases and proof by contradiction, suggesting that pretraining is not sufficient to teach the model to generate hypothetical subproofs. See Section 4.2.1 and Figure 4.
4. Model size does not strongly correlate with performance. Smaller (but not the smallest) models with instruction tuning and longer pretraining perform comparably to larger models.
<sup>1</sup>All analysis code, data, data generation scripts, and model outputs are publicly available at github.com/asaparov/prontoqa
Dataset	Automated evaluation of proofs	Contains proofs with multiple deduction rules	Tests proof depth generalization	Tests proof width generalization	Tests compositional generalization	Data generation code available
CLUTRR Sinha et al. [2019]	X	~	X	✓	~	✓
LogiQA Liu et al. [2020]	X	✓	~	~	~	human-annotated
ProofWriter Tafjord et al. [2021]	✓	~	✓	~	~	X
FOLIO Han et al. [2022]	X	✓	~	~	~	human-annotated
PRONTOQA Saparov and He [2023]	✓	X	✓	X	X	✓
PRONTOQA-OOD (this dataset)	✓	✓	✓	✓	✓	✓
TABLE 1: Comparison of existing datasets to evaluate reasoning ability. Datasets marked with  $\sim$  contain examples of varying width, depth, and compositionality, but these are not programmable (i.e. we cannot generate new examples controlling for these variables), and splitting the existing examples would produce highly imbalanced splits.
OD generalization of LLMs. Previous work has measured the generalization ability of LLMs on tasks such as bit parity and Boolean variable assignment [Anil et al., 2022], semantic parsing [Hosseini et al., 2022, Qiu et al., 2022], deductive reasoning [Zhang et al., 2022a, Sanyal et al., 2022, Kazemi et al., 2023], and arithmetic reasoning [Kudo et al., 2023], where the length/complexity of the test example is greater than that of the in-context examples. On the bit parity and variable assignment tasks, LLMs are able to generalize to longer inputs with scratchpad prompting [Nye et al., 2021], but this generalization is imperfect, and accuracy still degrades with increasing input length. Generally, larger models tend to be better at generalization than smaller ones. The studies on reasoning were limited to reasoning using modus ponens. Wu et al. [2021] tests the OOD generalization of transformers and graph neural networks on symbolic mathematical reasoning. Our study more systematically examines OOD generalization of LLMs to larger proofs as well as to other deduction rules.
Evaluating reasoning abilities of LLMs. A number of recent studies measured the reasoning ability of LLMs [Huang and Chang, 2022, Han et al., 2022]. Table 1 provides a comparison of our proposed dataset to datasets from these studies. Many of these datasets are not amenable to automated evaluation of proofs, relying instead on measuring label accuracy. The datasets also do not test for proof width generalization and compositional generalization. Some datasets focus on a limited set of deduction rules, namely modus ponens. Our work is closest to PRONTOQA [Saparov and He, 2023] but extends it to a complete set of deduction rules and to compositional proofs.
Understanding in-context learning. Recent work has shed some light on ICL, and the mechanism by which the model learns from in-context examples. Akyurek et al. [2023], Dai et al. [2023], von Oswald et al. [2022] showed that transformers can learn in-context by performing gradient descent on in-context examples internally. Xie et al. [2022], Wang et al. [2023] show that LLMs exhibit behavior similar to that of topic models where their output is dependent on a latent topic, and the in-context examples help to specify the topic. Ye et al. [2022] demonstrated that ICL is more effective when the in-context examples are both diverse and relevant to the test example. An et al. [2023] and Levy et al. [2022] explored the effect of in-context demonstrations on compositional generalization, showing that it benefits from diverse and individually simple demonstrations. Our results contribute to this growing literature by showing that the generalization behavior in ICL is different from that of supervised learning, and so algorithms such as gradient descent are not the only mechanisms underlying ICL.
A programmable dataset. Our main evaluation approach is to prompt the LLM with simpler proofs and test it on proofs with greater depth and width, or with those using additional deduction rules. Therefore, we require a programmable approach to data generation, where the deduction rules
Deduction rule	Formal definition	Natural language example
Implication elimination (i.e. modus ponens)	f(a) ∀x(f(x)→g(x)) g(a)	"Alex is a cat. All cats are carnivores. Alex is a carnivore."
Conjunction introduction	A B A ∧ B	"Alex is a cat. Alex is orange. Alex is a cat and orange."
Conjunction elimination	A ∧ B A	"Alex is a cat and orange. Alex is orange."
Disjunction introduction	A A ∨ B	"Alex is a cat. Alex is a cat or orange."
Disjunction elimination (i.e. proof by cases)	A ∨ B A ⊢ C B ⊢ C C	"Alex is a cat or a dog. Suppose Alex is a cat ... then Alex is warm-blooded. Suppose Alex is a dog ... then Alex is warm-blooded. Alex is warm-blooded."
Proof by contradiction	A ⊢ B ¬B ¬A	"Alex is cold-blooded. If Alex is a mammal, Alex is not cold-blooded. Suppose Alex is a mammal. Alex is not cold-blooded. This contradicts with Alex is cold-blooded. Alex is not a mammal."
TABLE 2: An overview of the deduction rules in PRONTOQA-OOD. The notation  $A \vdash B$  denotes entailment that  $B$  is provable from  $A$  .
"Alex is a dog. All dogs are mammals. Alex is a mammal. Alex is blue. All mean things are not blue. Suppose Alex is mean. Alex is not blue. This contradicts with Alex is blue. Therefore, Alex is not mean. Alex is a mammal and not mean."
FIGURE 2: An example of a compositional proof containing modus ponens, proof by contradiction, and conjunction introduction, shown in both natural language and a formal tree representation.
used, as well as the depth and width of each proof, are controllable parameters. To this end, we propose PRONTOQA-OOD, a generative process for synthetic reasoning questions. Each example in PRONTOQA-OOD contains a handful of premises, a query (the target fact to be proved/disproved), and a gold CoT containing the proof of the query. See Figure 10 for an example from this dataset. Specifically, we extend the PRONTOQA dataset [Saparov and He, 2023] that contains proofs generated from synthetic world models using modus ponens. (1) To evaluate reasoning using different deduction rules, we generate proofs with deduction rules for all connectives in propositional logic: conjunction  $\wedge$ , disjunction  $\vee$ , implication  $\rightarrow$ , and negation  $\neg$ . (2) To study width/depth generalization, the proof depth and width are controllable parameters in proof generation, where they control the number of generated deduction rules, and the number of premises in each deduction rule, respectively. (3) To study compositional generalization, we generate compositional proofs using a simple recursive procedure, where each proof contains multiple subproofs with distinct deduction rules.
Generating proofs with a complete set of deduction rules. We follow the deduction rules of natural deduction [Gentzen, 1935, Pfenning, 2004], a well-studied proof calculus with desirable completeness properties. $^{2}$  Examples of each deduction rule are shown in Table 2. For each type of deduction rule, we randomly generate a proof that applies that rule (see details in section A.4. An example of a compositional proof is shown in Figure 2. To prevent LLMs from exploiting knowledge from pretraining to solve the problems without reasoning, we use fictional names for all concepts (e.g. "wumpus" instead of "cat" etc.).
<sup>2</sup>With minor differences: Negation introduction/elimination, truth introduction, and falsity elimination are impossible to express in natural text, and they are omitted. We use proof by contradiction to reason over negation.
Varying proof width and depth. To characterize the size or complexity of each proof, we represent each proof as a tree (Figure 2), where each proof step corresponds to a node, and its premises correspond to the parent nodes. Then the size of the proof can be naturally described by the width and depth of this tree. When generating proofs, we control the depth by continuing to append proof steps until a proof of the desired depth is generated. The number of premises of deduction rules is set to the desired proof width.
Generating compositional proofs. To generate proofs combining many different types of deduction rules, we use a simple recursive procedure: (1) select a deduction rule uniformly at random, (2) select the premises for the selected rule, (3) recursively generate a subproof for each premise. See section A.5 for details and pseudocode.
Adding distractors. One key challenge to OOD generalization is shortcut solutions. For example, given the facts "Alex is a cat," "All cats are feline," since there is no other fact of the form "All cats are...," the model can deterministically follow the only valid deduction and conclude "Alex is feline." To make the heuristics uninformative, we add distractor sentences. In the above case, a distractor sentence would be "All cats are graceful." Then the model is forced to choose the correct premise from two options for the next deduction step. See Section A.7 for details on distractors for all deduction rules.
Formal evaluation of chain-of-thought. Unlike previous datasets that evaluate on a binary true/false answer, PRONTOQA-OOD requires LLMs to generate full proofs.<sup>3</sup> Therefore, we need a way to evaluate the correctness of the output proofs directly. The sentences in PRONTOQA-OOD are syntactically simple and amenable to semantic parsing, which allows us to formally analyze each step in the CoT. To determine whether a predicted CoT is correct, we: (1) semantically parse each CoT sentence into first-order logic, (2) determine whether each logical form follows from previous logical forms via a rule of deduction, and (3) compute whether there exists a path of correct steps from the premises to the goal. An example of this process is shown below:
"Alex is a dog.  
All dogs are mammals.  
Alex is a mammal."
Each proof step is considered correct if it is valid and if it immediately follows one of its premises. For further details see section A.6.
In this section, we test existing LLMs on PRONTOQA-OOD and analyze whether they can produce longer and compositional proofs given simpler demonstrations. We experiment with a variety of models, with different sizes and training objectives, as shown in Figure 3. In all experiments, we use 8-shot chain-of-thought prompting.
FLAN-T5	LLaMA	GPT-3.5	PaLM
Model Size	11B	65B	175B*	540B
Instruction Tuned	✓	×	✓	×
RLHF	×	×	✓	×
Access	Open	Limited	Limited	Limited
FIGURE 3: An overview and properties of the LLMs in our experiments. We place an asterisk* for GPT-3.5 since we were not able to verify its size.
We compare performance in two settings: (1) an in-demonstration (ID) setting where the 8 in-context demonstrations come from the same distribution as the test example, and (2) an out-of-demonstration (OOD) setting where the in-context demonstrations come from a distribution that is different from that of the test example.  $95\%$  confidence intervals are provided for all results.
True/false queries introduce an extra implicit negation step of the final conclusion. The negated conclusion may be tricky to represent in natural language given the extensive space of potential conclusions of the proofs that we generate.
<sup>4</sup>Note that if we only check for validity, the following proof would be considered correct: "Fae is a cat. All cats are carnivores. Fae is a carnivore. All carnivores are mammals. Fae is a mammal. All mammals are not herbivorous. Fae is not herbivorous." Here, the goal is to prove "Fae is not herbivorous." Every step of the proof is correct except "All mammals are not herbivorous." To avoid this, we require each step to immediately follow one of its premises.
$^{5}$ Following PRONTOQA, since adding further in-context examples did not improve performance.
FIGURE 4: (top) Proof accuracy across examples with different deduction rules. The in-context examples and test examples come from the same distribution. (bottom) Change in proof accuracy, where the test example is out-of-demonstration with respect to the in-context examples. That is, the test example has the specified deduction rule, but the in-context examples are uniformly distributed over all other deduction rules. See Figure 11 in the Appendix for the equivalent plot with absolute proof accuracy on the y-axis. See Figure 5 for an incorrect example. Implication elimination examples have proof width of 1 and depth of 2. Conjunction introduction, conjunction elimination, and disjunction introduction examples have proof width 3 and depth 2. Disjunction elimination examples have proof width 3 and depth 1. Proof by contradiction examples have proof width 2 and depth 1.
FIGURE 5: Example of an incorrect proof generated by GPT-3.5 on an out-of-demonstration disjunction elimination example. The premises (axioms) are given in blue, and invalid steps are given in red. For the full example, see Figure 14 in the Appendix.
Prove: Max is a torus.
Predicted answer: Max is a tumpus or a rompus or a lempus. Max is a tumpus.  
Tumpuses are wumpuses. Max is a wumpus. Rompuses are gorpuses. Max is a gorpus.  
Max is a gorpus.
Expected answer: Assume Max is a tumpus. Tumpuses are gorpuses. Max is a gorpus.
Assume Max is a rompus. Rompuses are gorpuses. Max is a gorpus.
Assume Max is a lempus. Lempuses are gorpuses. Max is a gorus.
Since Max is a tumpus or a rompus or a lempus, Max is a gorpus.
4.1 Can LLMs use deduction rules other than modus ponens?
We first evaluate whether LLMs "know" all deduction rules (Table 2) when provided with corresponding CoT prompts. For each deduction rule, we independently and identically generate 8 in-context examples and one test example, and prompt the model to answer the test example. We run each experiment for 100 trials and measure the accuracy of the output proofs. The accuracies are shown in the top chart of Figure 4. We emphasize that the  $\Delta$  proof accuracies in the bottom row of the figure should be interpreted in comparison with the accuracies in the top row (e.g. for some rules, the zero  $\Delta$  accuracy for FLAN-T5 is due to zero absolute accuracy). For clarity, we provide the same plots using absolute accuracy rather than  $\Delta$  accuracy in Section A.2.
In general, most models are able to use each deduction rule reasonably well, with GPT-3.5 performing the best. Similar to prior studies [Liang et al., 2022], we do not observe a strong correlation between model size and performance. LLaMA performs comparably to PaLM, despite being smaller. FLAN-T5 is smaller and performs reasonably on implication and conjunction elimination, but is not able to learn the other deduction rules.
FIGURE 6: (top-left) Proof accuracy on compositional examples where the in-context examples are also compositional examples with the same min depth and number of rule types. (bottom-left) Change in proof accuracy where the test examples are compositional but the in-context examples are those of individual deduction rules. See Figure 12 in the Appendix for the equivalent plot with absolute proof accuracy on the y-axis. (right) Example of an incorrect proof generated by GPT-3.5 on an out-of-demonstration example with min depth 2 and 4 rule types. The premises (axioms) are given in blue, and invalid steps are given in red. For the full example, see Figure 15 in the Appendix.
Prove: Polly is not a lempus.
Predicted answer: Polly is a wumpus, a jompus, and a tumpus. Everything that is a wumpus, a jompus, and a tumpus is not a lempus. Polly is not a lempus.
Expected answer: Polly is a tumpus. Polly is a jompus. Polly is a wumpus. Polly is a wumpus and a jompus and a tumpus. Everything that is a wumpus, a jompus, and a tumpus is not a lorpus. Polly is not a lorpus. Assume Polly is a lempus. Each lempus is an impus and a lorpus and a rompus. Polly is an impus and a lorpus and a rompus. Polly is a lorpus. This contradicts with Polly is not a lorpus. Polly is not a lempus.
4.2 Out-of-demonstration generalization
4.2.1 Can LLMs generalize to unseen deduction rules?
While the above results show that LLMs are able to reason with a variety of deduction rules, it is unclear whether the ability is learned from in-context examples or elicited from pretraining. We test the LLM with examples where the test proof requires a deduction rule that does not appear in the in-context examples (i.e. for each in-context example, we sample a deduction rule uniformly at random from the set of deduction rules excluding that of the test example). Our intuition was that LLMs would not be able to use deduction rules unless given explicit demonstrations thereof (aside from those like modus ponens which are well-represented in pretraining). The change in proof accuracy relative to the ID setting is shown in the bottom chart of Figure 4. Evidently, the models are able to use four deduction rules despite not being shown an in-context example with those rules: both conjunction rules, disjunction introduction, and (somewhat) implication elimination. GPT-3.5 was additionally able to use proof by contradiction by relying on an alternate deduction rule called modus tollens (i.e. given  $\neg f(c)$  and  $\forall x(g(c) \to f(c))$ , conclude  $\neg g(c)$ ). This is in contrast with McKenzie et al. [2022] which showed that reasoning with modus tollens exhibited inverse scaling behavior, and yet GPT-3.5 is able to use it correctly without any demonstrations. However, Wei et al. [2022a] has shown that when trained with additional compute, models are able to perform modus tollens. GPT-3.5 performed worse on disjunction elimination possibly due to the fact that there is no equivalent alternate rule (an example of an error is given in figure 5). However, no model is able to use disjunction elimination and proof by contradiction without demonstrations.
4.2.2 Can LLMs generalize to compositional proofs?
Next, we test whether the model is able to generalize to compositional proofs that contain multiple different deduction rules. In the ID setting, the in-context examples and test examples are both generated from the same distribution of compositional proofs. In the OOD setting, the in-context demonstrations contain non-compositional examples of each rule that appears in the test example. In Figure 6, in all but three experiments, we observe that the gap in proof accuracy between the ID and OOD settings is close to zero, indicating that the models are able to generalize compositionally