SlowGuess/NeurIPS_2023_OCR huggingface dataset:
FIGURE 7: (top row) Proof accuracy vs proof depth of test examples, where in-context examples have fixed proof depth 2. (bottom row) Proof accuracy vs proof width of test examples, where in-context examples have fixed proof width 2. Dashed lines indicate in-distribution accuracy, where the depth and width of the in-context examples are the same as that of the text-examples. In these experiments, there are 4 rather than 8 in-context examples.
to an extent. This is surprising since past studies show that LLMs struggle with compositional generalization, but this could be due to the fact that much of the previous work focused on semantic parsing rather than on reasoning. But there is prior work showing that models can generalize compositionally in some settings, such as in Hosseini et al. [2022] (see Figure 4) and in Press et al. [2022] (see Figure 6). In addition, our study is limited by the token limit of the LLMs, as we are not able to further increase the complexity of the proofs without reducing the number of in-context examples, which would render the results difficult to compare. GPT-3.5 and PALM have difficulty when the number of rule types is 4, with an example of an incorrect output given in the right side of Figure 6. Interestingly, PALM, LLAMA, and FLAN-T5 sometimes perform better in the OOD setting than in the ID setting, showing that, in ICL, it is not always best to provide demonstrations from the same distribution as the test example.
4.2.3 Can LLMs generalize to bigger proofs?
To test whether LLMs can generalize to bigger proofs, we test the models on examples where the proof width or depth is larger than those of the in-context examples. As is evident from Figure 7, when shown demonstrations of proofs of depth 2, the models' performance decreases with increasing depth. But this is due to the increase in the inherent difficulty of the task, as both ID and OOD accuracies decrease with increasing depth. Though the notable exception is GPT-3.5 on conjunction elimination, where ID accuracy remains high as OOD accuracy decreases. Models are able to generalize better with increasing proof width on conjunction elimination, possibly because there are ample examples of long conjunctions in natural language, but only GPT-3.5 is able to generalize to greater proof widths on conjunction introduction.
4.3 Do distractors help OOD generalization?
In supervised learning, one challenge to OOD generalization is spurious correlations [Zhang et al., 2022b]. Intuitively, if ICL were to behave like supervised learning on in-context examples [Akyurek et al., 2023, Dai et al., 2023, von Oswald et al., 2022], we would expect that without distractors, the models would overfit to the in-context examples and perform poorly on OOD examples. An example where distractors hurt generalization is shown in Figure 9 where GPT-3.5 copies many of the distractor sentences into the output, likely due to the fact that it has learned to apply a copying heuristic from the in-context demonstrations. It seems only GPT-3.5 acquires these heuristics in implication and disjunction elimination. Surprisingly, this is not the case for all deduction rules, as is
FIGURE 8: (top) Proof accuracy on examples where both the in-context examples and test examples have distractors. Sentences in all questions are ordered randomly. (middle) Proof accuracy when distractors are removed from the in-context examples, but not from the test examples. The sentences of the in-context questions have a fixed order (corresponding to a postorder traversal of the ontology tree), whereas the sentences in the test question have random order. See Figure 13 in the Appendix for the equivalent plot with absolute proof accuracy on the y-axis. (bottom) The same setting as (middle) except the distractors in the in-context examples are instead replaced with irrelevant sentences. Implication elimination examples have proof width of 1 and depth of 2. Conjunction introduction, conjunction elimination, and disjunction introduction examples have proof width 2 and depth 2. Disjunction elimination examples have proof width 3 and depth 1. Proof by contradiction examples have proof width 2 and depth 1.
Prove: Sally is dull.
Predicted answer: Sally is an impus. Impuses are sterpuses. Every grimpus is a zumpus. Every brimpus is a grimpus. Lorpuses are dumpuses. Brimpuses are vumpuses. Every lorpus is a brimpus. Sally is a brimpus. Each vumpus is opaque. Each dumpus is not brown. Every grimpus is dull. Sally is a grimpus. Sally is dull.
Expected answer: Sally is a brimpus. Every brimpus is a grimpus. Sally is a grimpus. Each grimpus is dull. Sally is dull.
FIGURE 9: Example of an incorrect proof generated by GPT-3.5 on an OOD implication elimination example where the in-context demonstrations have no distractors, but the test example does. The premises (axioms) are given in blue, and invalid steps are given in red. For the full example, see Figure 16 in the Appendix.
visible in Figure 8. The models' performance is largely unaffected, with the exception of a few rules for specific models. This is in stark contrast to supervised learning, where it is always best to train on examples from the same distribution as the test example.
In this study, we provide a systematic test of the general deductive reasoning capabilities of LLMs, specifically measuring their rule-, depth-, width-, and compositional generalization abilities. We found that LLMs exhibit mixed generalization to unseen deduction rules, but they exhibit more robust generalization to compositional proofs than previously suggested.
One important future direction is to better understand the mechanism of ICL and CoT prompting. We found that in many cases, for a given test example, the best in-context examples were drawn from a distribution distinct from that of the test example. This is not explained by existing theories of Bayesian inference [Xie et al., 2022] or gradient descent [Akyurek et al., 2023, Dai et al., 2023, von Oswald et al., 2022]. Are simpler examples better even if the test example is fairly complex? Should we include examples with a diverse set of deduction rules [Levy et al., 2022]? Or should the in-context examples focus on rules for which the model's OOD generalization is poor? Further study is needed to better characterize generalization from in-context examples.
Reproducibility statement
For the sake of reproducibility of the analysis, model outputs (except those of PaLM), code for data generation, and analysis code are freely available with a permissive open-source license at github.com/asaparov/prontoqa. The generated data for all experiments in this paper is available in the file generated_ood_data.zip. The command python makeplots.py produces all figures used in this paper. GPT-3.5 experiments were run using the OpenAI API with the model text-davinci-003 on April  $20^{th} - 23^{rd}$ , 2023. Experiments for Figure 7 and the bottom row of Figure 8 were run on August  $3^{rd} - 7^{th}$ , 2023.
We thank Tania Bedrax-Weiss, Xin Xu, and Deepak Ramachandran for their valuable feedback. This work was supported by Open Philanthropy, AWS AI, Samsung Advanced Institute of Technology (under the project Next Generation Deep Learning: From Pattern Recognition to AI), and in part through the NYU IT High Performance Computing resources, services, and staff expertise. NJ is supported by an NSF Graduate Research Fellowship under grant number 1839302.
Ekin Akyurek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? Investigations with linear models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=0gOX4H8yN4I.
Shengnan An, Zeqi Lin, Qiang Fu, Bei Chen, Nanning Zheng, Jian-Guang Lou, and Dongmei Zhang. How do in-context examples affect compositional generalization? CoRR, abs/2305.04835, 2023. URL https://arxiv.org/abs/2305.04835.
Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay V. Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. Exploring length generalization in large language models. CoRR, abs/2207.04901, 2022. doi: 10.48550/arXiv.2207.04901. URL https://doi.org/10.48550/arXiv.2207.04901.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan First, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. CoRR, abs/2204.02311, 2022. doi: 10.48550/arXiv.2204.02311. URL https://doi.org/10.48550/arXiv.2204.02311.
Honghua Zhang, Liunian Harold Li, Tao Meng, Kai-Wei Chang, and Guy Van den Broeck. On the paradox of learning to reason from data. CoRR, abs/2205.11502, 2022b. doi: 10.48550/arXiv.2205.11502. URL https://doi.org/10.48550/arXiv.2205.11502.
A.1 Example from PRONTOQA-OD
Q: Everything that is a lorpus, a brimpus, or a jompus is a shumpus. Every wumpus is a vumpus and a sterpus and a brimpus. Everything that is a vumpus, a grimpus, or a brimpus is a lempus. Everything that is a lempus or a jompus or a lorpus is a dumpus. Vumpuses are rompuses. Every sterpus is a gorpus. Everything that is a vumpus, a grimpus, or a brimpus is a dumpus. Wumpuses are shumpuses. Polly is a rompus. Polly is a wumpus. Prove: Polly is a lempus or an impus or a yumpus.
A: Polly is a wumpus. Every wumpus is a vumpus and a sterpus and a brimpus. Polly is a vumpus and a sterpus and a brimpus. Polly is a brimpus. Polly is a vumpus, a grimpus, or a brimpus. Everything that is a vumpus, a grimpus, or a brimpus is a lempus. Polly is a lempus. Polly is a lempus, an impus, or a yumpus.
FIGURE 10: An example from PRONTOQA-OOD. This is a compositional example with a min depth of 4 and 3 rule types. The given answer is the expected chain-of-thought. The question is shown in blue, the query in red, and the chain-of-thought/answer in green.
A.2 Results with absolute accuracy
FIGURE 11: (top) Proof accuracy across examples with different deduction rules. The in-context examples and test examples come from the same distribution. (bottom) Proof accuracy where the test example is out-of-demonstration with respect to the in-context examples (for comparison, the in-demonstration proof accuracy is shown as the dotted black bars). That is, the test example has the specified deduction rule, but the in-context examples are uniformly distributed over all other deduction rules. See Figure 5 for an incorrect example. Implication elimination examples have proof width of 1 and depth of 2. Conjunction introduction, conjunction elimination, and disjunction introduction examples have proof width 3 and depth 2. Disjunction elimination examples have proof width 3 and depth 1. Proof by contradiction examples have proof width 2 and depth 1.
FIGURE 12: (top) Proof accuracy on compositional examples where the in-context examples are also compositional examples with the same min depth and number of rule types. (bottom) Proof accuracy where the test examples are compositional but the in-context examples are those of individual deduction rules (for comparison, the in-demonstration proof accuracy is shown as the dotted black bars).
FIGURE 13: (top) Proof accuracy on examples where both the in-context examples and test examples have distractors. Sentences in all questions are ordered randomly. (middle) Proof accuracy when distractors are removed from the in-context examples, but not from the test examples (for comparison, the in-demonstration proof accuracy is shown as the dotted black bars). The sentences of the in-context questions have a fixed order (corresponding to a postorder traversal of the ontology tree), whereas the sentences in the test question have random order. (bottom) The same setting as (middle) except the distractors in the in-context examples are instead replaced with irrelevant sentences. Implication elimination examples have proof width of 1 and depth of 2. Conjunction introduction, conjunction elimination, and disjunction introduction examples have proof width 2 and depth 2. Disjunction elimination examples have proof width 3 and depth 1. Proof by contradiction examples have proof width 2 and depth 1.
A.3 Full examples of incorrect predicted proofs
Question: Lempuses are gorpuses. Rompuses are wumpuses. Tumpuses are wumpuses. Dumps are gorpuses. Every zumpus is a wumpus. Every brimpus is a gorpus. Yumpuses are gorpuses. Lempuses are wumpuses. Rompuses are gorpuses. Tumpuses are gorpuses. Max is a tumpus or a rompus or a lempus. Max is a zumpus, a tumpus, or a rompus.
Predicted answer: Max is a tumpus or a rompus or a lempus. Max is a tumpus.
Tumpuses are wumps. Max is a wump. Rompuses are gorpuses. Max is a gorpus.
Max is a corpus.
Expected answer: Assume Max is a tumpus. Tumpuses are gorpuses. Max is a gorus.
FIGURE 14: Example of an incorrect proof generated by GPT-3.5 on an out-of-demonstration disjunction elimination example. The premises (axioms) are given in blue, and invalid steps are given in red.
Question: Polly is a wumpus. Every rompus is a dumpus. Lempuses are vumpuses. Everything that is a wumpus, a jompus, and a tumpus is a dumpus. Polly is a gorpus. Every impus is a brimpus. Everything that is a wumpus, a gorpus, and a shumpus is a vumpus. Each lempus is an impus and a lorpus and a rompus. Everything that is a wumpus, a jompus, and a tumpus is not a lorpus. Zumpuses are lorpuses. Polly is a shumpus. Polly is a zumpus. Polly is a zumpus. Polly is a wumpus. Polly is a jompus. Polly is a tumpus.
Expected answer: Polly is a tumpus. Polly is a jompus. Polly is a wumpus. Polly is a wumpus and a jompus and a tumpus. Everything that is a wumpus, a jompus, and a tumpus is not a lorus. Polly is not a lorus.
Assume Polly is a lempus. Each lempus is an impus and a lorpus and a rompus. Polly is an impus and a lorpus and a rompus. Polly is a lorpus. This contradicts with Polly is not a lorpus. Polly is not a lempus.
FIGURE 15: Example of an incorrect proof generated by GPT-3.5 on an out-of-demonstration example with min depth 2 and 4 rule types. The premises (axioms) are given in blue, and invalid steps are given in red.
Question: Every grimpus is a zumpus. Each vumpus is opaque. Each dumpus is not brown. Each grimpus is dull. Lorpuses are moderate. Every brimpus is rainy. Impuses are temperate. Lorpuses are dumpuses. Brimpuses are vumpuses. Every lorpus is a brimpus. Zumpuses are angry. Grimpuses are yumpuses. Every brimpus is a grimpus. Impuses are sterpuses. Sally is an impus. Sally is a brimpus.
FIGURE 16: Example of an incorrect proof generated by GPT-3.5 on an OOD implication elimination example where the in-context demonstrations have no distractors, but the test example does. The premises (axioms) are given in blue, and invalid steps are given in red.
A.4 Generative process details
In this section, we describe the process to generate examples of each deduction rule.
Implication elimination (i.e. modus ponens) Given  $f(c)$  and  $\forall x(f(x) \to g(x))$ , prove  $g(c)$ . These are the examples in the original PRONTOQA. We follow the same process here:
1. Generate an ontology. For simplicity, we generate linear ontologies, consisting of a collection of concepts, as well as subtype-supertype relations between those concepts (i.e. concept  $f$  is a subtype of the supertype  $g$  if every instance of  $f$  is an instance of  $g$ ). For simplicity, we limit each type to have at most one supertype.
2. Perform a random walk of length  $k$  from a randomly selected start vertex, where  $k$  is the desired proof depth.
3. Traverse the edges of the ontology and convert each into a sentence of the question.
4. Convert each step of the random walk into a sentence of the gold chain-of-thought.
Note that this process allows us to generate proofs of any depth, but the width is fixed to 1.
Conjunction introduction Given  $A$  and  $B$ , prove  $A \land B$ . The generative process is a modified version of that for implication elimination. Instead of generating rules of the form  $\forall x (f(x) \to g(x))$ , we generate rules of the form  $\forall x (f_1(x) \land \ldots \land f_n(x) \to g(x))$ , where  $n$  is the proof width. Given,  $f_1(c), \ldots,$  and  $f_n(c)$ , the model must first prove  $f_1(c) \land \ldots \land f_n(c)$  before applying implication elimination to prove  $g(c)$ . To increase the depth of the proof,  $g(c)$  itself can be part of a conjunct in the antecedent of another rule.
Conjunction elimination Given  $A \land B$ , prove  $A$ . These examples are identical to those in conjunction introduction, except the conjunction appears in the consequent of each rule, rather than in the antecedent:  $\forall x(f(x) \to g_1(x) \land \ldots \land g_n(x))$  where  $n$  is the proof width.
Disjunction introduction Given  $A$ , prove  $A \lor B$ . These examples are identical to those in conjunction introduction, except the conjunction is replaced with disjunction:  $\forall x(f_{1}(x) \lor \ldots \lor f_{n}(x) \to g(x))$  where  $n$  is the proof width. But note that grounded axioms are not necessary for every disjunct: To apply the rule  $\forall x(f_{1}(x) \lor \ldots \lor f_{n}(x) \to g(x))$ , knowing  $f_{n}(c)$  is sufficient, and we do not need to generate grounded axioms for the other disjuncts  $f_{i}(c)$  for  $i < n$ .
Disjunction elimination (i.e. proof by cases) Given  $A_1 \lor \ldots \lor A_n$ , and  $A_i \vdash C$  for all  $i$ , prove  $C$ . Here,  $n$  is the proof width. While it is possible to construct proofs containing multiple nested applications of disjunction elimination, such proofs are quite complex, even for humans to understand, and so we fix the depth of these examples to 1. To generate an example, we first generate the disjunction:  $f_1(c) \lor \ldots \lor f_n(c)$ . Next, generate the rules for each case:  $\forall x (f_i(x) \to g(x))$  for all  $i$ . The goal is to prove  $g(c)$ .
Proof by contradiction Given  $A \vdash B$  and  $\neg B$ , prove  $\neg A$ . Note that this is a rule composed of two natural deduction rules: negation elimination and introduction. But since those individual rules do not lend themselves to a natural text representation, we choose to study their composition. Similar to disjunction elimination, it is possible to construct proofs containing multiple nested applications of proof by contradiction, but such proofs are unnaturally complex. So we fix the depth to 1. To generate an example, we first generate an axiom  $\neg g(c)$ . Next, for each subproof, we generate a rule  $\forall x(f_1(x) \lor \ldots \lor f_n(x) \to g(x))$ , where  $n$  is the proof width. The goal is to prove  $\neg f_1(c) \land \ldots \land \neg f_n(c)$ . Note that in addition to proof by contradiction, this proof requires disjunction introduction, implication elimination, and conjunction introduction.
Note that the above list constitutes a complete set of deduction rules from propositional natural deduction, save for one rule: implication introduction. However, it is unclear how to construct an example with this deduction rule where its difficulty can be controlled by increasing the width or depth of the proof (e.g. how can a statement of the form  $A_1 \to A_2 \to \ldots \to A_n$  be expressed in natural language?).
Algorithm 1: Pseudocode for generating examples of compositional proofs in PRONTOQA-OOD. In this algorithm,  $\Omega$  denotes the set of all logical forms. The function generate_compositional-proof is initially called with parameters  $\Omega$ ,  $\varnothing$ ,  $d$ ,  $e$ , and false, where  $d$  is the requested depth and  $e$  is a randomly selected entity name (e.g. alex, fae, etc). sample is a helper function that, given an input set of logical forms  $S$  and an entity  $e$ , returns sample.uniform(\{set of logical forms in  $S$  with minimal depth where all atoms are of the form  $t(e)$  where  $t$  is a predicate\}).
function generate_compositional-proof (set of possible conclusions (logical forms)  $C$  disallowed deduction rules R, requested depth d, ground entity e, is proof hypothetical h)   
initialize  $A$  as the set of all deduction rules excluding those in  $R$  /\* filter deduction rules such that: (1) an element of  $C$  can be a conclusion of the rule, (2) for which we have sufficient depth, and (3) we don't create overly complex logical forms \*/   
if  $C$  does not contain a conjunction set  $A = A\backslash \{$  conjunction_introduction}   
if  $C$  does not contain a disjunction set  $A = A\backslash \{$  disjunction_introduction}   
if  $h =$  true or  $d = 1$  or  $C$  does not contain a negation set  $A = A\backslash \{$  proof_by_confliction}   
if  $h =$  true or  $d = 1$  or  $C$  contains only conjunctions or only disjunctions set  $A = A\backslash \{$  disjunction eliminatio}   
if  $C$  contains only conjunctions or only disjunctions set  $A = A\backslash \{$  conjunction eliminatio}   
if  $C$  contains only conjunctions or only disjunctions and any operand is negated set  $A = A\backslash \{$  implication eliminatio}   
if  $d = 0$  or  $C$  contains a singleton logical form or  $A = \emptyset$  return axiom step with conclusion given by sample(C,e)   
 $r =$  sample_uniform(A)   
if  $r =$  implication eliminatio do for any  $c\in C,a$  and  $c$  share any operands or negations of operands while  $a =$  generate_compositional-proof(  $\Omega ,\varnothing ,d - 1,e,h)$  do a and s do not share any operands or negations of operands while  $s =$  sample(C,e) return implication eliminatio with premises a and  $\forall x(a[e\to x]\to s[e\to x])$    
else if  $r =$  conjunction_introduction initialize  $P$  as an empty list, and  $i = 0$ $L = |C|$  if  $C$  contains only conjunctions, else  $L = 3$  do let  $C_i = i^{th}$  operand of  $C$  if  $C$  contains only conjunctions, else  $C_i = \Omega$  a= generate_compositional-proof(Ci,{conjunction eliminatio},d-1,e,h) if a is atomic and a is not any other operand of C append a to P i=i+1 while i< L return conjunction_introduction with premises P   
else if  $r =$  conjunction eliminatio let  $C^{\prime}$  be the set of conjunctions of length 3,  $i =$  sample_uniform({1,2,3})  $C^\prime = \{c\in C^\prime :$  the  $i^{th}$  operand of  $c^{\prime}$  is in C do a= generate_compositional-proof(C', {conjunction_introduction},d-1,e,h) while a has no duplicate operands, and each operand of a is not itself a conjunction or disjunction return conjunction eliminatio with premise a and conclusion given by the  $i^{th}$  operand of a