SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Names: We assign a name (from a list of manually written names) to two of the players and then write rules in the form of If [PLAYER i] has a name that starts with the same letter as [PLAYER 2], then ...
- Jobs: We manually write a list of pairs of jobs and the industry they belong to. Examples include (nurse, healthcare), (high school teacher, education), and (sales manager, marketing). We use the job in the fact and the industry in the rule body. The model has to know which job is part of which industry.
- Volume: The facts mention that one of the players has an object (a notebook or a ball) and give the dimensions of the object (the height and width for notebook and the radius or diameter for the ball). The rule body asks whether the object fit in a box with some given dimensions. The model has to understand how 3D objects fit inside each other to be able to connect the fact to the rule.
- Events: We manually write a list of world events and the year when they occurred. Examples include (world war 1 started, 1914), (the first man landed on moon, 1969) and (Obama's presidency started, 2009). Then we write facts of the form The [PLAYER] is watching a movie from [YEAR] and rules of the form If the [PLAYER] is watching a movie that was released [before/after] [EVENT], then .... The model has to know the time for major world events to be able to connect the fact and the rule.
- Friends: We first generate a positive integer  $x$  corresponding to the number of friends a player has. Then, we either generate a fact such as The [PLAYER] has  $x$  friends or The [PLAYER] has  $x_1$  friends that are [ADJECTIVE] and  $x_2$  that are not where  $x_1 + x_2 = x$ . Then we decide if we want to use a more than or less than relation. In the former case, we generate a number  $y < x$  and in the latter case  $y > x$ . Then we generate a rule of the form If the [PLAYER] has [more than/less than]  $y$  friends, then ....
Due to the nature of the extra knowledge and reasoning cases we consider, we only add such cases at the last theory generation step of Algorithm 1 (i.e. when  $d = 1$ ); Otherwise, we will need to follow
Facts: The reindeer has three friends that are energetic and 1 friend that is not. The reindeer will turn 18 months old in a few minutes. The finch does not build a power plant near the green fields of the reindeer.
Rules: Rule1: For the reindeer, if the belief is that the german shepherd disarms the reindeer and the finch builds a power plant near the green fields of the reindeer, then you can add that "the reindeer is not going to acquire a photograph of the finch" to your conclusions. Rule2: The leopard trades one of its pieces with the cobra whenever at least one animal builds a power plant close to the green fields of the finch. Rule3: Regarding the reindeer, if it is more than five years old, then we can conclude that it acquires a photo of the finch. Rule4: Here is an important piece of information about the reindeer: if it has fewer than 5 friends then it acquires a photo of the finch for sure.
Preferences: Rule3 is preferred over Rule1. Rule4 is preferred over Rule1.
Question: Does the leopard trade one of its pieces with the cobra?
Prediction: Since [...] and according to Rule1 [...] and Rule1 is preferred over Rule3 so the reindeer is not going to acquire a photograph of the finch. [...]
Figure 21: An example of a wrong proof generated by PaLM 540B (fewshot) where the error is due to misunderstanding a preference.
Facts: The liger has a card that is violet in color. The worm invests in the company whose owner is the mouse. The goat does not create one castle for the pigeon.
Rules: Rule3: The living creature that does not create a castle for the pigeon will never hug the liger. [...]
Preferences: Rule2 is preferred over Rule1. Rule6 is preferred over Rule5.
Question: Does the tiger invest in the company whose owner is the mermaid?
Prediction: [...] We know the goat does not hug the tiger, and according to Rule3 "if the goat does not hug the tiger, then the tiger does not hug the tiger"
Figure 22: An example of a wrong proof generated by T5 where the error is due to hallucinating facts and rules.
generating sub-questions and rules for questions such as The dog is named Paco leading to unnatural rules such as "If ... then the dog is named Paco.
C.3 Entities, Predicates, and Templates
Table 3 presents the set of entities, predicates, and templates used in BoardgameQA. To make the problem slightly more challenging in terms of language complexity, we use different entities, predicates and templates in the test set.
C.4 Sample Proofs
In Figure 26, we provide some sample proofs with depths 1, 2, and 3 from the dataset. It can be observed that as the depth of the dataset increases, the length and difficulty of the proof increases substantially.
Facts: The llama creates one castle for the camel. The llama has a card that is white in color. The monkey is a public relations specialist.
Rules: [...] Rule2: Regarding the llama, if it has a card whose color appears in the flag of Italy, then we can conclude that it does not suspect the truthfulness of the mermaid. Rule3: If the monkey manages to convince the mermaid, then the mermaid is not going to pay money to the seahorse. Rule4: For the mermaid, if the belief is that the dinosaur dances with the mermaid and the llama does not suspect the truthfulness of the mermaid, then you can add "the mermaid pays some $$ to the seahorse" to your conclusions. Rule5: If the monkey works in marketing, then the monkey manages to convince the mermaid.
Preferences: Rule1 is preferred over Rule2. Rule4 is preferred over Rule3.
Question: Does the mermaid pay money to the seahorse?
Prediction: We know the llama has a card that is white in color, white appears in the flag of Italy, and according to Rule2 [...]
Facts: The otter has a 10 x 18 inches notebook. The otter is 15 months old. The duck does not tear down the castle that belongs to the otter.
Rules: Rule1: If the otter does not enjoy the companionship of the dugong, then the dugong captures the king (i.e. the most important piece) of the bear. Rule2: Regarding the otter, if it is more than four years old, then we can conclude that it does not want to see the dugong. Rule3: Regarding the otter, if it has a notebook that fits in a 18.1 x 12.8 inches box, then we can conclude that it does not want to see the dugong.
Preferences: None
Question: Does the dugong capture the king of the bear?
Prediction: Since the otter is 15 months old and 15 months is less than 4 years and according to Rule2 if the otter is less than 4 years old then it does not want to see the dugong so the otter does not want to see the dugong. [...]
Figure 23: An example of a wrong proof generated by T5 where the model got distracted and ended up on a wrong proof path.
Figure 24: An example of a wrong proof generated by PaLM 540B (fewshot) where the error is due to changing a rule in such a way that a proof can be found, when a proof does not exist.
Figure 25: A sample of theory and question generation from Algorithm 1. Initially, the question has been selected to be (dog, attack, cat). The input depth  $D = 1$  indicates that a theory with one hop of reasoning should be generated. Then a fact (dog, unite, lion) and a rule  $R1$ : (X, unite, lion)  $\Rightarrow$  (X, attack, cat) have been generated. Notice that the combination of the fact and the rule conclude (dog, attack, cat). Next we randomly decide if a conflict should be generated. The decision is yes, so we generate another fact (dog, respect, cat) and rule (dog, respect, cat)  $\Rightarrow !$  (dog, attack, cat). Notice that the two rules have contradictory conclusions now. We next decide randomly the type of the conflict, and Type2 is selected in this case. Therefore, we add  $R2 > R1$  to our preferences, remove one of the facts generated for the conflicting rule, and make recursive calls for the remaining facts, which is only (dog, unite, lion). This call is made with  $D = 0$ , therefore the stopping criterion triggers and we add (dog, unite, lion) to our set of facts.
Sample Depth 1 Proof: We know the dog has a card that is white in color, white starts with "w", and according to Rule2 "if the dog has a card whose color starts with the letter "w", then the dog prepares armor for the eel", and Rule2 has a higher preference than the conflicting rules (Rule1), so we can conclude "the dog prepares armor for the eel". So the statement "the dog prepares armor for the eel" is proved and the answer is "yes".
Sample Depth 2 Proof: We know the whale eats the food of the koala, and according to Rule2 "if something eats the food of the koala, then it rolls the dice for the leopard", so we can conclude "the whale rolls the dice for the leopard". We know the buffalo does not know the defensive plans of the cat, and according to Rule1 "if something does not know the defensive plans of the cat, then it doesn't attack the green fields whose owner is the leopard", so we can conclude "the buffalo does not attack the green fields whose owner is the leopard". We know the buffalo does not attack the green fields whose owner is the leopard and the whale rolls the dice for the leopard, and according to Rule4 "if the buffalo does not attack the green fields whose owner is the leopard but the whale rolls the dice for the leopard, then the leopard raises a peace flag for the zander", and for the conflicting and higher priority rule Rule3 we cannot prove the antecedent "the leopard does not respect the kudu", so we can conclude "the leopard raises a peace flag for the zander". So the statement "the leopard raises a peace flag for the zander" is proved and the answer is "yes".
Sample Depth 3 Proof: We know the crocodile is named Milo and the donkey is named Meadow, both names start with "M", and according to Rule11 "if the crocodile has a name whose first letter is the same as the first letter of the donkey's name, then the crocodile steels five points from the cockroach", and for the conflicting and higher priority rule Rule4 we cannot prove the antecedent "at least one animal knows the defensive plans of the buffalo", so we can conclude "the crocodile steals five points from the cockroach". We know the crocodile steels five points from the cockroach, and according to Rule6 "if something steals five points from the cockroach, then it does not remove from the board one of the pieces of the tiger", so we can conclude "the crocodile does not remove from the board one of the pieces of the tiger". We know the aardvark proceeds to the spot right after the lion, and according to Rule8 "if something proceeds to the spot right after the lion, then it does not eat the food of the spider", and for the conflicting and higher priority rule Rule3 we cannot prove the antecedent "at least one animal attacks the green fields whose owner is the catfish", so we can conclude "the aardvark does not eat the food of the spider". We know the aardvark does not hold the same number of points as the phoenix, and according to Rule7 "if something does not hold the same number of points as the phoenix, then it doesn't give a magnifier to the eagle", so we can conclude "the aardvark does not give a magnifier to the eagle". We know the aardvark does not give a magnifier to the eagle and the aardvark does not eat the food of the spider, and according to Rule2 "if something does not give a magnifier to the eagle and does not eat the food of the spider, then it knows the defensive plans of the tiger", so we can conclude "the aardvark knows the defensive plans of the tiger". We know the aardvark knows the defensive plans of the tiger and the crocodile does not remove from the board one of the pieces of the tiger, and according to Rule10 "if the aardvark knows the defensive plans of the tiger but the crocodile does not remove from the board one of the pieces of the tiger, then the tiger becomes an enemy of the cheetah", so we can conclude "the tiger becomes an enemy of the cheetah". So the statement "the tiger becomes an enemy of the cheetah" is proved and the answer is "yes".
Figure 26: Sample proofs from the dataset at depths 1, 2 and 3. Higher depth proofs are substantially longer than the lower depth proofs.
Table 3: Categories, descriptions, and examples of incomplete information in BoardgameQA. For lexical entailment, world knowledge, event times, and affordance, a list of examples is written manually from which the sampling procedure can select. In the other cases, examples are generated automatically.