SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Examples of Model Failures: To showcase some of the reasons why models struggle with the BoardgameQA dataset, in Figure 11 13 we show some example proofs where the model made a variety of errors and in Figures 15-24 we provide specific examples where the model generated wrong proofs. Some of the dominant error cases (showcased in the examples) include: 1- hallucinating or misunderstanding conflicts and preferences, 2- not being able to correctly fill in the incomplete information, 3- misunderstanding logical rules, 4- failing to prove both elements in a conjunction, 5- getting distracted by distracting facts and rules and going on a wrong proof path, and 6- being unfaithful to the provided facts and rules and changing them so a proof can be found in the case where no proof exists.
Confusion Matrices: The confusion matrices for the model predictions with respect to the golden labels on the BoardgameQA dataset is provided in Figure 14 for various depths. One interesting observation is that for models tuned with proofs, while the model performs well at predicting unknown labels in lower depths, in higher depths they tend to generate proofs (with proved or disproved labels) even when the label is unknown (e.g., for examples with label unknown in depth 1 the prompt-tuned model predicts unknown for 268 instances, while in depth 2 it predicts unknown only for 2 instances). This may be because when the search space for a proof increases, LMs cannot verify all possible solutions and decide that the label is unknown. Instead, they start a path in the hopes that it ends in a proof.
Standard Deviations: Due to the high cost of experimenting with large language models, the results in the paper are reported based on a single run. To gain insight into the standard deviation of the
<sup>3</sup>In this case, we set  $p_{Conf} = 1.0$  for the first call we make to Algorithm 1; Otherwise, the dataset will have a spurious correlation that can be exploited without doing any reasoning.
Depth 2 Example: A few players are playing a boardgame. The current state of the game is as follows. The gorilla is watching a movie from 1952. The woodpecker has a football with a radius of 28 inches, and swears to the duck. The zebra swims in the pool next to the house of the butterfly. And the rules of the game are as follows. Rule1: The gorilla will not enjoy the company of the vampire if it (the gorilla) works in healthcare. Rule2: The gorilla enjoys the company of the vampire whenever at least one animal swims inside the pool located beside the house of the butterfly. Rule3: If there is evidence that one animal, no matter which one, acquires a photograph of the peafowl, then the woodpecker calls the vampire undoubtedly. Rule4: Here is an important piece of information about the woodpecker: if it has a football that fits in a  $58.4 \times$  61.5 x 59.6 inches box then it does not call the vampire for sure. Rule5: If at least one animal leaves the houses that are occupied by the dragon, then the vampire does not disarm the bulldog. Rule6: Here is an important piece of information about the gorilla: if it is watching a movie that was released after the first man landed on moon then it does not enjoy the companionship of the vampire for sure. Rule7: If something swears to the duck, then it leaves the houses that are occupied by the dragon, too. Rule1 is preferred over Rule2. Rule3 is preferred over Rule4. Rule6 is preferred over Rule2. Based on the game state and the rules and preferences, does the vampire disarm the bulldog?
Model Output: Since the woodpecker has a football with a radius of 28 inches and according to Rule4 if the woodpecker has a football that fits in a  $58.4 \times 61.5 \times 59.6$  inches box then it does not call the vampire so the woodpecker does not call the vampire. Since the woodpecker does not call the vampire and according to Rule3 if there is evidence that one animal, no matter which one, acquires a photograph of the peafowl then the woodpecker calls the vampire so the woodpecker does not acquire a photograph of the peafowl. Since the woodpecker does not acquire a photograph of the peafowl and according to Rule5 if at least one animal leaves the houses that are occupied by the dragon then the vampire does not disarm the bulldog so the vampire disarms the bulldog. So the statement the vampire disarms the bulldog is proved and the answer is "yes".
- Since Rule4 activates, the model assumes the woodpecker does not call the vampire without checking whether Rule3 (the higher priority rule) activates or not. Luckily for the model, though, Rule3 does not activate.
- While Rule4 indeed activates, the model makes a planning error by starting from Rule4. The correct proof plan is to start with Rule7 followed by Rule5
- An initially wrong proof plan leads to consequent wrong steps in the proof planning (Rule4 is followed by Rule3).
The model misunderstands the antecedent and the consequent of Rule3.
The model jumps to Rule5 and arrives at a conclusion about the question, without proving the antecedent of Rule5.
The model misunderstands the sign of the consequent of Rule5.
Figure 12: Highlighting some of the model errors for a depth 2 example when using PaLM 540B.
Depth 3 Example: A few players are playing a boardgame. The current state of the game is as follows. The butterfly swears to the wolf. The cougar has 8 friends, and is named Chickpea. The crow is named Charlie. The dove has a basketball with a diameter of 15 inches, and is named Tessa. The dove has a card that is white in color. The monkey is named Mojo. The shark is named Paco. The worm is named Tango. The husky does not tear down the castle that belongs to the finch. And the rules of the game are as follows. Rule1: From observing that an animal brings an oil tank for the bison, one can conclude the following: that animal does not smile at the beetle. Rule2: If the frog refuses to help the dalmatian, then the dalmatian is not going to invest in the company owned by the beetle. Rule3: Are you certain that one of the animals manages to convince the ostrich and also at the same time smiles at the beetle? Then you can also be certain that the same animal negotiates a deal with the badger. Rule4: If the monkey has a card whose color appears in the flag of Netherlands, then the monkey does not hide the cards that she has from the pelikan. Rule5: The pelikan unquestionably shouts at the badger, in the case where the monkey does not hide her cards from the pelikan. Rule6: If the monkey has a name whose first letter is the same as the first letter of the shark's name, then the monkey does not hide her cards from the pelikan. Rule7: Here is an important piece of information about the cougar: if it has a name whose first letter is the same as the first letter of the crow's name then it manages to convince the ostrich for sure. Rule8: Regarding the dove, if it has a name whose first letter is the same as the first letter of the worm's name, then we can conclude that it enjoys the companionship of the dalmatian. Rule9: If the cougar negotiates a deal with the badger and the pelikan shouts at the badger, then the badger falls on a square of the mouse. Rule10: One of the rules of the game is that if the dove enjoys the companionship of the dalmatian, then the dalmatian will, without hesitation, invest in the company whose owner is the beetle. Rule11: The cougar will smile at the beetle if it (the cougar) has more than seven friends. Rule12: This is a basic rule: if the finch negotiates a deal with the pelikan, then the conclusion that the pelikan will not shout at the badger follows immediately and effectively. Rule13: This is a basic rule: if the husky does not tear down the castle that belongs to the finch, then the conclusion that the finch negotiates a deal with the pelikan follows immediately and effectively. Rule1 is preferred over Rule11. Rule2 is preferred over Rule10. Rule5 is preferred over Rule12. Based on the game state and the rules and preferences, does the badger fall on a square of the mouse?
Model Output: Since the cougar has 8 friends and according to Rule11 if the cougar has more than seven friends then the cougar will smile at the beetle so the cougar smiles at the beetle. Since the cougar smiles at the beetle and according to Rule2 if the cougar smiles at the beetle then the cougar does not negotiate a deal with the badger so the cougar does not negotiate a deal with the badger. Since the cougar does not negotiate a deal with the badger and according to Rule9 if the cougar negotiates a deal with the badger and the pelikan shouts at the badger then the badger falls on a square of the mouse so the badger does not fall on a square of the mouse. So the statement the badger falls on a square of the mouse is disproved and the answer is "no".
- Since Rule11 activates, the model assumes the cougar will smile at the beetle without checking whether Rule1 (the higher priority rule) activates or not. Luckily for the model, though, Rule1 does not activate.
The model hallucinates a non-existing rule instead of Rule2 which makes it shortcut to a conclusion.
The model fails to understand conjunction and assumes Rule9 activates by only showing the first antecedent activates.
The model misunderstands the sign of the consequent of Rule9.
Figure 13: Highlighting some of the model errors for a depth 3 example when using PaLM 540B.
results, we trained the BERT model (our cheapest baseline) 10 times on each of the datasets from the main text and computed the standard deviations. The standard deviations ranged from 0.3 to 2.6.
We conducted our experiments on v3 TPUs for all the models, except for the 540B models where we used v4 TPUs due to their larger size. All the experiments were done using the T5X framework [38] available at https://github.com/google-research/t5x.
For the fewshot experiments, to ensure the demonstrations and the question fit in the prompt, due to the large size of the examples in BoardgameQA, we only included one example per label as demonstration (i.e. 3 examples one with label proved, one with disproved, and one with unknown in the case of 3-way classification datasets). For each example in the test set, we selected the demonstrations randomly from the training set, while ensuring that they contain both types of conflict resolution. For the prompt-tuning experiments, we used a prompt-size of 100 in all experiments, as it worked best in the experiments of [26]. We allowed a maximum of 50K training steps with a batch size of 8, a learning rate of 0.1, and a weight decay rate of 0.0001. We evaluated the model after each 1000 steps on the validation set and selected the checkpoint with the best validation accuracy for testing. For the finetuning experiments, we also set the batch size to 8 but set the learning rate to
(a) Finetuned w/o proofs (BERT Large)
(b) Finetuned w/o proofs (BERT Large)
(c) Finetuned w/o proofs (BERT Large)
(d) Finetuned w/ proofs (T5 XXL)
(e) Finetuned w/ proofs (T5 XXL)
(f) Finetuned w/ proofs (T5 XXL)
(g) Prompt-tuned w/ CoT (PaLM 62B)
(h) Prompt-tuned w/ CoT (PaLM 62B)
(i) Prompt-tuned w/ CoT (PaLM 62B)
(j) Fewshot w/ CoT (PaLM 540B)
(k) Fewshot w/ CoT (PaLM 540B)
(1) Fewshot w/ CoT (PaLM 540B)
Figure 14: Confusion matrices for various models on the BoardgameQA dataset.
0.001 and allowed 50K epochs (since it required more time to converge) and reported the results for the epoch that achieved best validation accuracy.
Facts: The seahorse has a basketball with a diameter of 27 inches, and is currently in Istanbul. The wolf does not take over the emperor of the snake. And the rules of the game are as follows.
Rules: [...] Rule2: If the seahorse has a basketball that fits in a  $37.1 \times 37.4 \times 23.9$  inches box, then the seahorse surrenders to the swan. Rule3: The seahorse will surrender to the swan if it (the seahorse) is in Turkey at the moment. [...]
Preferences: Rule5 is preferred over Rule4.
Question: Does the swan swear to the woodpecker?
Prediction: Since the seahorse is in Istanbul and according to Rule3 if the seahorse is in Turkey at the moment then the seahorse surrenders to the swan so the seahorse surrenders to the swan. Since the seahorse surrenders to the swan and according to Rule2 if the seahorse has a basketball that fits in a  $37.1 \times 37.4 \times 23.9$  inches box then the seahorse surrenders to the swan so the seahorse has a basketball that fits in a  $37.1 \times 37.4 \times 23.9$  inches box. [...]
Figure 15: An example of a wrong proof generated by PaLM 540B (fewshot) where the error is due to misunderstanding a logical rule (given a fact  $f$  and a rule  $f' \to f$  the model concludes that  $f'$  must be true.
Facts: The mule dances with the fish. [...]
Rules: [...] Rule2: There exists an animal which dances with the fish? Then, the dachshund definitely does not capture the king (i.e. the most important piece) of the owl. Rule3: If something does not capture the king (i.e. the most important piece) of the owl and additionally not disarm the zebra, then it disarms the woodpecker. [...]
Preferences: Rule1 is preferred over Rule5. Rule3 is preferred over Rule4.
Question: Does the dachshund disarm the woodpecker?
Prediction: [...] and according to Rule4 [...] and for the conflicting and higher priority rule Rule3 we cannot prove the antecedent "the dachshund does not capture the king of the owl", [...].
Figure 16: An example of a wrong proof generated by PaLM 62B (prompt-tuned) where the error is due to assuming an antecedent of a high priority rule cannot be proved, whereas it can indeed be proved.
C BoardgameQA Details
Here, we provide more in depth details about the generation and properties of the BoardgameQA dataset. A sample of theory and question generation from Algorithm 1 is provided in Figure 25.
C.1 Consistency of the Dataset
A defeasible theory is called consistent if whenever a conflict arises, the preferences can be used to resolve the conflict. In BoardgameQA, we aim to produce consistent theories. To avoid inconsistencies and loops, each time we call the function in Algorithm 1, we only allow it to sample from the entities that have not been used in other rules and (sub-)questions. As an example, if we have a rule such as  $a \land b \Rightarrow c$ , then when we call the algorithm for  $a$  and  $b$  recursively, we use separate entities in the facts and rules produced for the sub-branch of  $a$  and for the sub-branch of  $b$ . This way, we ensure that when we derive new facts in the sub-branch of  $a$ , it does not defeat some of the derivations in the sub-branch of  $b$  (and the derivations in the later stages of the proof do not defeat the earlier derivations). That is because the set of facts used in the rule bodies are for separate entities, and are therefore disjoint. We also apply defeasible reasoning on the final logical theory to ensure that the question can be derived from the theory.
C.2 Incomplete Information
Here, we provide more information about the nature and type of incomplete information in BoardgameQA.
- Age: We first generate a positive integer  $x$  corresponding to the age of one of the players expressed in days. Then we decide if we want to use a more than or less than relationship. In the case of the former, we next generate another integer  $y < x$  and in the case of the latter  $y > x$ . Then, we randomly decide a target unit (days, weeks, months, or years) for each integer and convert them to that unit. Let  $x'$  and  $y'$  represent the obtained values measured with the new units. Then we add a fact [PLAYER] is  $x'$  [unit] old and a rule if the [PLAYER] is [more than/less than]  $y'$  [unit] old then .... The model has to be able to convert units of time and then compare them.
Figure 17: An example of a wrong proof generated by PaLM 62B (prompt-tuned) where the error is due to filling the missing information incorrectly.
Facts: The cougar manages to convince the ostrich, and smiles at the beetle. The dalmatian is named Chickpea. The pelikan has 8 friends. The seal is named Charlie.
Rules: Rule1: If something smiles at the beetle and manages to convince the ostrich, then it negotiates a deal with the badger. Rule2: If the dalmatian has a name whose first letter is the same as the first letter of the seal's name, then the dalmatian invests in the company whose owner is the beetle. [...] Rule4: From observing that an animal brings an oil tank for the shark, one can conclude the following: that animal does not shout at the badger. Rule5: If the cougar negotiates a deal with the badger and the pelikan shouts at the badger, then the badger falls on a square of the mouse.
Preferences: Rule4 is preferred over Rule3.
Question: Does the badger fall on a square of the mouse?
Prediction: Since the cougar manages to convince the ostrich and smiles at the beetle and according to Rule1 [...] so the cougar negotiates a deal with the badger. Since the cougar negotiates a deal with the badger and according to Rule5 if the cougar negotiates a deal with the badger and the pelikan shouts at the badger then the badger falls on a square of the mouse and Rule5 is preferred over the conflicting and lower priority rule Rule4 so the badger falls on a square of the mouse. So the statement the badger falls on a square of the mouse is proved and the answer is "yes".
Figure 18: An example of a wrong proof (but correct label) generated by PaLM 540B (fewshot) where the error is due to 1- failing to prove one element of the conjunction and also identifying a non-existence conflict between two rules.
- Affordance: We manually wrote object properties/affordances, and a list of items that have those properties. Examples of properties include sharp, drink, and music and examples of objects for each of these properties include knife, cappuccino, and flute respectively. The facts are of the form The [PLAYER] has a [OBJECT] and the rules are of the form If the [PLAYER] has [AN OBJECT WITH PROPERTY] then .... The model has to know about the object properties to connect the facts and the rules.
- Colors: We first identify groups of colors based on some property. This includes being a primary color, being a rainbow color, and being a color in the flag of country  $X$ . Then, we generate facts of the form The [PLAYER] has a card which is [COLOR] in color and rules of the form If the [PLAYER] has a card whose color is [COLOR PROPERTY] then .... The model has to have information about colors to connect the facts and the rules.
- Money: We first generate a positive integer  $x$  corresponding to the amount of money a player has, then we randomly decide if we want the comparison to be between two or three players. We also decide if we want to use a more than or less than relation. If the comparison is between two players and more than is used, then we generate another integer  $y < x$  and if less than is used then  $y > x$ ; if the comparison is between three players and more than is used then we generate  $y, z$  such that  $y + z < x$ , and if less than is used then  $y + z > x$ . We then generate facts of the form The [PLAYER i] has  $x$  dollars. and rules of the form If [PLAYER i] has [more than/less than] than [PLAYER j] and [PLAYER k] combined, then .... The model has to do a summation and decide which quantity is more or less.
- Textual Entailment: We manually write multiple pairs of sentences where one implies the other. Examples include (assassinated the mayor, killed the mayor), (struggles to find food, has difficulty to find food), and (purchased a luxury aircraft, owns a luxury aircraft). The first element of the pair is used in the fact and the second element in the body of a rule. The model has to identify the entailment.
- Places: We manually write a list of cities and the countries they are located in. The city names are used in the facts (The [PLAYER] is in [CITY] right now.) and the countries are used in the rule bodies (If the [PLAYER] is in [COUNTRY] right now, then .... The model has to know which city is in which country.
Facts: The monkey has a card that is orange in color. The monkey has a football with a radius of 29 inches, and is a farm worker.
Rules: [...] Rule4: If the monkey has a basketball that fits in a  $26.3 \times 25.6 \times 24.2$  inches box, then the monkey swims in the pool next to the house of the crab. [...]
Preferences: Rule5 is preferred over Rule1.
Question: Does the monkey pay money to the rhino?
Prediction: Since the monkey has a football with a radius of 29 inches and according to Rule4 if the monkey has a basketball that fits in a  $26.3 \times 25.6 \times 24.2$  inches box then the monkey swims in the pool next to the house of the crab [...]
Figure 19: An example of a wrong proof generated by PaLM 540B (fewshot) where the error is due to 1- mistaking a football with a basketball, and 2- not being able to fill in the incomplete information by realizing that a ball with a radius of 29 inches does not fit in a  $26.3 \times 25.6 \times 24.2$  inches box.
Facts: The gorilla is watching a movie from 1952. The woodpecker has a football with a radius of 28 inches, and swears to the duck. The zebra swims in the pool next to the house of the butterfly.
Rules: Rule1: The gorilla will not enjoy the company of the vampire if it (the gorilla) works in healthcare. [...] Rule4: Here is an important piece of information about the woodpecker: if it has a football that fits in a 58.4 x 61.5 x 59.6 inches box then it does not call the vampire for sure. Rule5: If at least one animal leaves the houses that are occupied by the dragon, then the vampire does not disarm the bulldog. [...] Rule7: If something swears to the duck, then it leaves the houses that are occupied by the dragon, too.
Preferences: Rule1 is preferred over Rule2. Rule3 is preferred over Rule4. Rule6 is preferred over Rule2.
Question: Does the vampire disarm the bulldog?
Prediction: Since the woodpecker has a football with a radius of 28 inches and according to Rule4 if the woodpecker has a football that fits in a  $58.4 \times 61.5 \times 59.6$  inches box then it does not call the vampire so the woodpecker does not call the vampire. [...]
Figure 20: An example of a wrong proof generated by PaLM 540B (fewshot) where the error is due to starting with a distracting fact that took the proof on a wrong path (the correct proof is to first use the fact The woodpecker swears to the duck and Rule7 to conclude that The woodpecker leaves the houses occupied by the dragon, and then use Rule5 to conclude that The vampire does not disarm the bulldog.