BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Information
Authors: Mehran Kazemi
Quan Yuan
Deepti Bhatia
Najoung Kim
Xin Xu
Vaiva Imbrasaite
Deepak Ramachandran
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: natural language reasoning
defeasible reasoning
large language models
TL;DR: We create a dataset for natural language reasoning when the provided information may be contradictory, and then benchmark various types of language models and show that current models perform poorly on this task.

Abstract: Automated reasoning with unstructured natural text is a key requirement for many potential applications of NLP and for developing robust AI systems. Recently, Language Models (LMs) have demonstrated complex reasoning capacities even without any finetuning. However, existing evaluation for automated reasoning assumes access to a consistent and coherent set of information over which models reason. When reasoning in the real-world, the available information is frequently inconsistent or contradictory, and therefore models need to be equipped with a strategy to resolve such conflicts when they arise. One widely-applicable way of resolving conflicts is to impose preferences over information sources (e.g., based on source credibility or information recency) and adopt the source with higher preference. In this paper, we formulate the problem of reasoning with contradictory information guided by preferences over sources as the classical problem of defeasible reasoning, and develop a dataset called BoardgameQA for measuring the reasoning capacity of LMs in this setting. BoardgameQA also incorporates reasoning with implicit background knowledge, to better reflect reasoning problems in downstream applications. We benchmark various LMs on BoardgameQA and the results reveal a significant gap in the reasoning capacity of state-of-the-art LMs on this problem, showing that reasoning with conflicting information does not surface out-of-the-box in LMs. While performance can be improved with finetuning, it nevertheless remains poor.

BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Information
Mehran Kazemi, Quan Yuan, Deepti Bhatia, Najoung Kim, Xin Xu,
Vaiva Imbrasaite, Deepak Ramachandran
Google Research
{mehrankazemi, yquan, bhatiad, njkim, xxujasmine, vimbrasaite, @anahhandradd@gogglee.. com com
Abstract
Automated reasoning with unstructured natural text is a key requirement for many potential applications of NLP and for developing robust AI systems. Recently, Language Models (LMs) have demonstrated complex reasoning capacities even without any finetuning. However, existing evaluation for automated reasoning assumes access to consistent and coherent set of information over which models reason. When reasoning in the real-world, the available information is frequently inconsistent or contradictory, and therefore models need to he equipped with a strategy resolve such conflicts when they arise. One -idely-applicable way of resolving conflicts is to impose preferences over information sources (e.g., based on source credibility information recency) and adopt the source with higher preference. In this paper, we formulate the problem of reasoning with contradictory information guided by preferences over sources as the classical problem of defeasible reasoning, and develop a dataset called BoardgameQA for measuring the reasoning capacity of LMS this setting. BoardgameQA also incorporates reasoning with implicit background knowledge, to better reflect reasoning problems downstream applications. We benchmark various LMS on BoardgameQA and the results reveal significant gap in the reasoning capacity state-of-the-ar LMs on this problem, showing that reasoning with conflicting information does not surface out-of-the-box in LMs. While performance can be improved with finetuning, it nevertheless remains poor.
Introduction
fundamental goal of Al since early days has been automatically applying logical or deductive
reasoning to draw new conclusions from existing knowledge [29, 20]. Since large amount of knowledge is available in the form of natural language, tremendous effort has been put into developing models that understand and reason over natural language [23, 42, 54, 32, 12, 57] (see [35] for survey). Recent years have seen substantial improvements in this direction thanks to advancements in pretrained language models (LMs) [8, 9] that handle unstructured data more flexibly, combined with advanced prompting techniques [52, 31], and modular reasoning approaches [23, 12).
Existing work in automated reasoning in natural language usually assumes that the provided knowl- edge is consistent and reliable. But many applications, the collection of information one has to reason with is inconsistent and contradictory. This is the case, for instance, when reasoning performed with information found in different online sources or social media (e.g., retrieval- augmented LMs [17, 3J). When input sources contradictory, gies to resolve the contradictions. One simple and practical formulation, which we adopt in this work, is to resolve the conflicts based preferences over the information sources: when con-
consider various strate-
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

flict arises, the information from the source with a higher preference should be used to solve the reasoning problem. Depending on the application, preferences can be assigned based on dif- ferent criteria, e.g., based on the credibility of websites or social media users, or based on the recency of the information with newer information being preferred over older information. Ex- ceptions to generics can also be expressed as preferences; for example, generic knowledge such as "birds fly" (see also [6]) should be overridden by exceptions such as "penguins are birds but do not fty" (see also [IJ) when reasoning about penguins. Figure demonstrates an example of reasoning problem with conflicting information, where the conflict is resolved based on recency.
Reasoning with conflicting information guided by preferences can be formulated as a form of the classical defeasible reasoning problem [33, 19, 28]. In this work, we study the rea- soning ability of LMs in this setting. Toward this goal, we create synthetic dataset where each example contains a defeasible theory (a set of input facts, possibly contradictory rules, and preferences over the rules), and a question about that theory. Answering the questions in the dataset requires multi-hop reasoning and conffict resolution over the input theory. The difficulty level (e.g., the depth, amount and type of conflicts, etc.) of the examples in the dataset can be controlled automatically, enabling targeted comparisons of various aspects of reasoning.
Facts: Fiona wants to travel from country X lo country' Y and stay for 5 days. Does Fiona need to book cavid19 test appointment?
Info, Source (date: 2022-04-20)
Info, Source (date: 2022 04-27)
All traveiers entering Y from
Travelers visiting Y for less than month do not require covid19
need show negative covid19
resulls.
anymore.
Ressoning: Fiona traveling from X and wants
for lees than two sourcee
sources apply. conclusions confict, but source 2 may be preferred since 8 is more recent. So Fiona doeen't need a covid leel, and so does not need to book an appointment.
Figure 1: A reasoning problem with contradictory information (conflict resolved based on recency).
We also note that while a large number of logical reasoning benchmarks provide all the knowledge needed to answer questions [48, 41, 42, 18], such benchmarks do not reflect common real-world scenarios where implicit background knowledge plays an important role in reasoning. Moreover, models that translate the textual examples into logical form and then leverage off-the- solvers may excel on these datasets, which does not reflect the true performance of such models in real- world applications. For these reasons, in BoardgameQA only part of the knowledge required to solve the problem is provided as input to the LM; the missing knowledge has to come from the LM itself.
The problems in our dataset are formulated as scenarios of a board game, hence we name it BoardgameQA! ! A board game theme allows us to create synthetic scenarios with complex de- feasible rules to reason about that seem natural when stated in text and hence allows background commonsense world knowledge to also be used. To the best of our knowledge, BoardgameQA is the first dataset for multi-hop reasoning with contradiclory inputs. Figure 2 shows sample example from the dataset where the conflict resolution and missing knowledge have been highlighted.
We benchmark various LMs on BoardgameQA and measure their defeasible reasoning capacity. Most notably, our results reveal that LMs perform poorly when reasoning with conflicting sources, especially in the few-shot setting (compared to the finetuning setting) suggesting that preference understanding and defeasible reasoning capacities do not surface out-of-the-box in pretrained LMs. Secondly, we find that smaller LMs perform poorly when not all of the required information is provided as input. These results highlight a critical gap the reasoning capacity of current LMs, considering that reasoning over contradicting and incomplete sets of information is a common scenario in many applications, and is key for developing robust AI systems.
2 Related Work
Our work spans three dimensions: 1- text-based logical reasoning, 2- reasoning with conflicting sources, and 3- reasoning with incomplete information. In the following section, we briefly summarize the literature on each of these axes that relate to our work.
1 All dataset variations used in our experiments can be found at: https://storage: http://orrrgggoggleepp.... com/ gresearch/ wwwah//aaaaaea/aaaaaaaaaee also be found at https:// ations under attps://storag: googleaps com/ beoseearchhBoardgaanee (including the ones men- tioned above) available under the CC BY license.
zip. variation with depth 4 (not used our experiments) can googleapis com/ gresear wwwsseachh/ooarggaeel// wetptee///d/eptddaeeeeeeee hepp//.pthh... All vari-
2

---

Facts: few players are playing boardgame. Here the current state of the game. The dog has $3. The lon has $48. The frog has 81 dollars, and has knife.
Rules: frog more money than llon the dog combined, then frog builds a power plant close to grean of
the cat. R2: somothing attacks the grean flakss of the cal, thon does not build a power plant near the graon flolds of the cat. [-]
Proferences: R2 preferred over R1. [-]]
Question:
frog build power plant near green fields the cal?
Proof: We know the frog has $81, 81 is more than 31-8-51 which the lotal money of the dog and lion combired, and according to R1
I-.), and for the conficciig and higher prirrty rule R2 we cannot prove the antecedent "the frog antacks the green hields of the calt, so statement "tee frog buils power plant the
conclude "the frog build power plant close the fields of the cat". green of
proved the answer is "yes'.
sample example from BoardgameQA that requires one hop of reasoning. The text in violet highlights conflict resolution and the text in blue highlights the missing information.
Figure 2:
Text-based logical reasoning approaches: Earlier works on natural language logical reasoning have finetuned LMs to directly provide answers to logical reasoning questions [11, 4, 40, 18). Later work showed that explicitly generating the entire proof leads to substantial improvements both in the case of finetuming and in the case of few-shot learning [31, 13, 58, 60]. In addition. modular reasoning approaches where the LM is used as a tool within a reasoning algorithm [23, 12, 51, 24] have been shown to achieve both performance gains and more precise intermediate proof chains. In this paper, we experiment with four types of approaches: 1- finetuning without explicit reasoning steps, 2- finetuning with explicit reasoning steps, 3- prompt- tuning with chain-of-thought (Co'T) prompting [52], and 4- few-shot in-context learning with COT.
Text- logical reasoning datasets: Many datasets have been created to measure the logical reasoning ability of NLP models [48, 42, 61, 18, 45]. In Table I, we provide a comparison of (a subset of) these datasets along three desired features in this work. All datasets compared contain only facts and rules that are non-comtee wow.-oootaaiicingg....mmm The datasets closest to our work are CREPE [30], FalseQA [21] and ConditionallA [47]: the first two provide false pre-suppositions in the question which can be considered as statements that contradict the ground truth, and in last one the answers to the questions follow a "If X then yes, if Y then no" format.
Reasoning with confticts: From the early days of AI, reasoning with conflicting information has been an important topic and many approaches have been developed to handle such conflicts [34, 33, 37]. The problem we study in this paper is an instance of defeasible reasoning [33, 19, 28] which has applications in various domains (especially in legal reasoning) [43, 16. 7] and has been argued to be one of the most important future directions a recent survey of LM reasoning literature [56]. In defeasible reasoning, there are preferences over the rules and in the case of conffict between two rules, the conclusion from the higher preference rule is accepted. Previous work on defcasible reasoning with natural language has studied the problem of adjusting the probability of conclusion based on new (single-hop) evidence [39, 27). Our work extends this line of work by developing a dataset for multi-hop defeasible reasoning with preferences over sources.
Reasoning with incomplete information:
Several existing reasoning benchmarks adopt a setup
where part of the required information is missing and needs to come from the model itself [46, 5, 2, 50]. Some datasets also employ a setup in which none of the required rules are provided as input [49, 15, 45, 22]. Our work focuses mainly cases where part of the knowledge needs to come from the model and another part of the knowledge is provided as input.
Background and Notation
We let & (ei, -.. eN) and P (pi,.... ... .... represent set of entities and predicates. We represent a fact in the logical form using the triple notation (ei, ex), where e,, e& E & and p5 E P. and rule as r rb rh where rb represents the body of the rule and rı represents the head. We use I to indicate negation. A monotonic theory T (F,R) is a tuple containing a set F of (positive or negative) facts, and a set R {ri..... ... (TT of rules. We let T F f represent that the fact f can be derived from the theory T using the standard inference rules of logic (See Shoenfield [44]), For a monotonic theory T (F, R), i = f. then for any theory T' such that T' (F U F", R), we also have T' F ff (that is, adding new facts does not change previously derived facts).
Defeasible Theory: A defeasible theory ( = (F,R,O) is triple containing set F of facts, set R fri,...., ||R of rules, and set O {rre Tla ........ . r'a) of pair-wise relative
3

---

Dataset
we bAbl 15
CLUTRR
FOLIO
Proot
Writer
PrOnto OOD
AR-LSAT
ENWN
Leep of Thought
CREPE
Falce
QA
Conditional QA
Boardgame
QA
Feature
Contradictory
Inf lnformstion
x
x
x
x
X
X
X
X
Incomplete
x
Information
Auto.
Diff.
Control
x
Table I: A comparison of BoardgameQA with some of the widely-used logical reasoning datasets
(bAbI 15 [53], CLUTRR [45], FOLIO [18]. ProofWriter [48], PrOntoQA-OOD) [42], AR-LSAT [61]. ENWN [46], leap-of-thought [50], CREPE [30], FalseQA [21]. and ConditionalQA [47]) in terms of three key features. We use ~ in the case of incomplete information for CLUTRR because there is only a fixed set of information that needs to come from the model. in the case of automatic difficulty control for leap-of-thoughs because the depth of reasoning fixed (difficulty added through distractors), in the of contradictory information for CREPE and FalseQA because the contradiction is with the ground truth not with another source of information, and in the case of contradictory information for ConditionalQA because while the answer question be yes under one set of conditions and no under another set of conditions, the two sets of conditions are mutually exclusive.
priorities/preferrnnes between rules.? The rules hold defeasibly, meaning the conclusion from a rule may be defeated by contrary evidence from higher priority rule. This happens, for example, when one rule implies something is true but another rule with a higher priority implies is false; in such cases, we accept the conclusion from the higher priority rule (see Figure 1). We let ( f represent that can be derived from defeasible theory ((d) after resolving conflicts. Note that the initial facts F are internally consistent and always have priority over the derived facts. We assume the theory is defeasibly consistent, meaning whenever a conflict arises, the preferences can be used to resolve it. An example of a defeasible theory ( is as follows:
Example 3.1. F (Tweety a penguin. }, R {ri Penguins are birds. T2 Birds fly. T3 : Penguins do not fly. O = {r'- > T2). .. From the theory, one can first use Tı to derive that "Tweety is a bird". Then, one can use 12 to derive that "Tweety flies". However, one can also use T3 to derive that "Tweety does not fly", which is in conftict with the previous derivations. Since accept the derivation that "Tweety does not fly".
Conflict types: Conflicts can arise for rules whose heads cannot be simultaneously true, e.g., for two rules r (a) r has higher priority than and we can prove rb. and (b) r has lower priority than r' and we can prove r3 but we cannot prove rg. In the first case, one does not need to take into account rgg for conflict resolution, but in the second case it is critical take ró into account. We name the first type of conflict Typel conflict and the second type Type2.
and r' rgg -t1z. For theory (()) with these two rules, T F= in two cases:
4 The BoardgameQA Dataset
We now describe how we create dataset for measuring the ability of LMs in reasoning with conflicting inputs defeasible setup. Our dataset creation follows a backward story generation strategy similar to [55, 23J. Each example in the dataset contains a (defeasible) theory T and question q. The goal is predict whether (()) FFr 9. or ( F= lg. neither. Therefore, the label space for each question is (proved, disproved, unknown). We next describe how we generate examples with the label proved; examples with the label disproved unknown created by modifying the examples with label proved.
The facts of each theory describe the current state of board game, the rules of each theory represent the rules of the board game, and the questions are about the game. the design of BoardgameQA, we include several variables that can be used sample examples with varying levels of difficulty with respect to several finer- amer-grained properties (c.g., depth, number and types of conflicts).
Note: Many types of preferences be converted into pair-wise relative preferences.

---

Calegory
Descripton
Example Facts
Example Fulle
Conver-
Comparee the age an entity ceriain age specilled with different unite.
dog is 13 months and a halt old
dog more than year
sion
The dog named Paco. named Pashmak.
that starts with the same letter as the name of the cat, then
the dog
Orthography
Asks about the letters names.
Numeric
Some numbers are required
summee and then compared to other numbers.
The dog has two frionds that are nice and five that are not
the dog has less
friends,
Comparisons
then
ment
Levical Entail
but the fact entails the rule bady.
and rule body are identical
The dog assassineted the mayor
If the dag killed the mayor, then
World Knowl
adgi
needed to cornect the fact to the rula body.
Some knowlodge about the world
The dog currentty Mentreal.
the dog is currently in Canada, then
Knowledge sbout
of events needed
The dog wetching movie
If dog watching movie that relsased after Covid19 started, then ...
Evant
to connect the fact to the rule body.
was released in 2006.
The and rule body have part of
The rula relation. cortain festurelaftordance of the fact.
dog
dog works healthccre, then ...
Affordance
dog has
the dog a sharp object,
Knowlodge of what objects what other objects required.
The dog ball with radius of the ball that in 28 35 inchos,
Volumes
15 inches.
Table 2: Categories, descriptions, and examples of incomplete information in BoardgameQA. For lex- ical entailment, world knowledge, event times, and affordance, list of examples is written manually from which the sampling procedure can select. In others, examples are generated automatically.
Algorithm 1 GenerateTheory
Entities and predicates: We start with a predefined set of entities & (c.g., dog, cat, lion, etc.) and predefined set of predicates P (e.g., invite for dinner; at- tack the fields, etc.) that sample from to generate facts and rules. We use the animals as entities and the boardgame- inspired verbs/operations as our predicates. Using these entities and predicates, we can create facts such as the dog attacks the fields of the lion. To make the problem more challenging, we use different entities and predicates across training and test sim- ilar to [25]. The full list of entities and predicates is provided in Appendix C.3. 13:
Input: Question q. Depth d
1: if d == then
2:
add(oFacts(Q)
3: else
4: Q, Ruue
5: addToRules(r)
6:
if CoinFlip(pcou)) == Conflict then
7:
Andd audToRules(r")
Q',
8:
9:
if CoinFliip(Com//tppp Typel then
10:
Q = Q SubSample(
11:
add[ToPreeereccestrr."" else
12:
Q RewoReonnSuuuuusstiiiiinn Q')
14:
add/oorrefereccestr" r)
Rule types: We adopt a set of 6 rule templates containing existential and uni- versal quantifiers, conjunctions, and miss- ing information. The rules are as fol- lows: 1- VX : (X,Pi,e) + (X, P2,22.)), 2- VX (X, (2,p2,62) +
15:
for qi in Q do
16:
GenerateTheoryice d-1)
Algorithm 2 SampleRuleAnddSugg
ex), 3- (e1,PI, E2) + (e2, ea), 4- (61, P1, A 62) = (s,,Ps, ea),
Input: Question q
I: r SampleQuestionn
5- (,) = (2).P. and 6- ex((X)I,e1 = (e2.P2.C) C3), where X represents a universally or existentially bounded variable, each e; represents an entity, and each pj represents a predicate. The fifth rule template corresponds to a rule where the predicate (or object entity) in the rule body may not be an element of P (resp. E). For more information, see below.
2: if r is rule with incomplete info (type 5) then
3: Sample Q {91,.., 9n3 and Q {41, ... gmm
s.t. q can be derived from @ and r, and @ can be derived from Q
4: else
5: Sample Q = {91, ..., 9np s.t. q can be derived from @ and r.
6:
return Q. r
To generate each ex- ample, we first sample a question es are sampled from E and pj is sampled from P. We also sample the sign of the question (positive or negative). For example, we might sample the question !ídog, atluck the fields, lion) asking whether the dog does not attack the fields of the lion. The question then converted into natural language using a template (see Appendix C.3).
Selecting a question:
(e,, Pj. ex) that should be proved or disproved, where and
5

---

Theory generation: The theory generation is the main component of the dataset generation that constructs the facts, rules and question to be used in each example. A high-level description is provided in Algorithm 1 and an example generation is shown in Appendix C. We first sample some sub-questions Q = (91.....9vv) and rule r which has Q in its body and q in its head, such that q can be derived from Q and r. The sampling is done by first selecting one of the aforementioned rule types, then matching the head of the rule to the question q. and then sampling sub-questions Q based on the body of the rule. For example for the question !(dog, attack the fields, lion), we might sample the first rule type (see the six types above), then p2 will be mapped to attack the fields and 62 will be mapped to lion, and we also sample sub-question such (dog, unite with, cat) and add the rule VX : (X, unite with, cat) =I(X, attacks the fields, lion) to our set of rules. We then make a recursive call for each 9; to generate new rules and facts for them.
We then decide whether a conflict should be introduced or not, by using a biased coin flip with PCoff representing the probability of conflict. If the decision is to produce conflicts, then we generate another set of sub-questions O' = ....... yn and another rule r' such that !q can be derived from Q and r'. Then we probabilistically decide if we want to generate a Type1 or a Type2 conflict using a biased coin flip with probability PConfTpet. If the first case is selected, then r > r' is added to the preferences. In this case, we can make recursive calls for all or subset of the facts in Q'. Otherwise, r' > r is added to the preferences. In this case, we make recursive calls for all but one of the facts in Q' (selecting randomly) to ensure that r' does not activate.
Proofs: We keep track of the facts, rules, and preferences during the generation process and turn them into proofs for the examples.
Stopping criterion:
Every time we make recursive call to the function in Algorithm 1, the example will contain one extra hop in its proof. We set the stopping criterion as the number of hops in the proof. Toward this goal, we included an argument in Algorithm which corresponds the target maximum number of hops in the proof; d decreases by one every time we make a recursive call. When the algorithm is called with d 0, instead of generating rules and sub-questions for the input question q. we simply add q to our set of facts.
We generate examples with incomplete information where part of the knowledge should come from the LM (corresponds to rule type 5). For a question in the the- ory generation phase, we sample sub-questions Q and rule r such that å can be derived based on Q and y can be derived from I and r. We then hide å from the model so the model has to derive it itself. Algorithm 2 describes the procedure. We use a separate body of world knowledge, commonsense knowledge, mathematical, and orthography reasoning for generating Q and Q (see Table 2 for high-level description and Appendix C.2 for more details). For example, for the goal "the dog unites with the cat" we generate the sub-question "The dog is in Montreal." and the rule "If the dog is in Canada, then the dog unites with the cat.". Then, an extra reasoning step is needed from the model to recognize that Montreal in Canada.
Incomplete information:
We generate sub-questions and rules that require extra knowledge and reasoning with probability PMisulfei other- wise, we create sub-questions and rules that require no extra knowledge and reasoning. To make the problem more chal- lenging, we only include some categories of extra knowl- edge and reasoning in the training set; this ensures that the models cannot simply learn the extra knowledge from the training set and use it in the test set.
1x
Log(Count)
6 6x
5 5x
ProatViriter
4x100
Pronto
BoardgamcOA
Average Example Length
Average Unique Tokens
A comparison of BoardgameQA with ProofWriter [48] and PrOntoQA [41] in terms of average length of examples and average number of unique tokens per example on depth 3 of the datasets.
Figure 3:
Conversion to natural language: Finally, once gen- erate the facts, rules, preferences, and question, we use manually constructed templates turn each of them into a textual format. To make the problem more challenging, we use multiple templates per rule type and use some of the templates only in the test set (see Appendix C.3 for details). comparison of BoardgameQA with other prominent deductive reasoning datasets in terms of the average length of examples and the average number of unique tokens per example is provided in Figure 3.

---

60
Depth
Hajonty Class
Accurbo
00
[BERT Larget
(T5 XXX3
IPaLM 528)
(TILM 8281
IPALM 542B)
O'LAN PALl 5403)
Figure 4: The model performances on depths 1-3 of the BoardgameQA dataset. Many models struggle on this dataset, especially with higher depths.
Disproved and unknown examples: So far, we described how to generate examples with the label proved. Generating examples with the label disproved can be done simply by first generating an example with the label proved and then negating the question. Also, generating examples with the label unknown can be done by perturbing the theory until the statement the question cannot be derived from the theory (e.g., reducing the amount of money of the frog to 50 dollars in the example of Figure 2). We randomly select and apply the following perturbations to the theory and run defeasible solver implemented based on the scalable solver in [28] on the resulting theory until the label becomes unknown: change the predicate of fact or rule, 2- change the sign of a fact or an element of the rule, 3- replace fact with a new fact, and 4- flip the order of preference.
Experiments
One of the primary goals of our experiments is to verify if LMs are capable of reasoning in a defeasible setup. For this reason, we conduct experiments with various LM architectures (encoder-only, encoder- decoder, and decoder-only) and various pre-training and learning paradigms (finetune with and without proofs, prompt tuning, few-shot in-context learning. and instruction-tunelt Specifically, we test 1) finetuning BERT-large [14] with classification head to predict the label directly. 2) finetuning T5 1.1 XXL [36] to generate the entire proof and then the label, 3) few-shotting PaL. 62B and PaLM 540B [9] where provide demonstration examples and chain-of-thought (CoT) in the prompt (the CoT corresponds to the proof), 4) few-shotting the instruction- -inettued FLAN-PaLM 540B [10] with CoT, and 5) soft prompt-tuning [26] PaLM 62B with CoT where instead of providing a static prompt, we make the prompt embedding learnable and tune its parameters using the training data (the rest of the LM parameters frozen). We report classification accuracy the metric. We also report the majority class baseline (~33% since our labels are balanced).
Dataset sizes:
To gain more detailed understanding of the models' defeasible reasoning capacity, we create several variations of BoardgameQA. The nature the variation will be discussed in the remainder of this section with each experiment. For each variation, we sample 1000 examples for train, 500 for validation, and 1000 for test. We sample an equal number examples from each label.
5.1 Can LMs Reason with Contradictory Inputs?
As explained in Section 4, Boardgame makes use of number of variables that control various aspects of the dataset such as the amount and types conflict and the amount of extra knowledge required. We start by creating default version of the dataset that exhibits each of these properties to some degree by setting PCwy/ = 0.5, PCon/Typel = 0.5, and PMisstaf = 0.5. We then generate three datasets with depth 1-3 (i.c., requiring 1-3 hop(s) of reasoning, respectively), and measure the performance of our baselines on these datasets.
The results are in Figure 4. The tuned models perform reasonably on depth 1, but their performance substantially degrades on depths 2-3. This contrasts with previous observations for monotonic reasoning (e.g.. in [11, 48]) where finetuned LMs reach near-perfect performance even higher depths. This indicates that reasoning with contradictory inputs is more difficult even with finetuning. Moreover, we see that the few-shot models perform poorly across all depths showing that conflict resolution not achieved out-of-the-bov with pretrained models. This includes both PaLM and instruction
finetuned enstructio-finetuned FLAN PaLM models. PaLM 540B performs better than PaLM 62B showing that
7

---

larger models may have higher capacity for defeasible reasoning. More insights from full confusion matrices can be found in Appendix A.
Hereafter, due to inference costs, we only experiment with finetuned BERT and T3, prompt-tuned PaLM 62B, and few-shot PaLM 540B, and with examples of depth 2 to keep a medium level of difficulty in terms of reasoning hops and enable measuring the effect of the other factors.
5.2 Does Correct Label Prediction Mean Correct Proof?
Recently, has been shown that although large LMs achieve high accuracy on label prediction for (monotonic) reasoning task, they do so by generating spurious proofs that do not represent valid steps of reasoning [23]. There is also evidence that LMs frequently exploit spurious correlations the data distribution to achieve high label accuracy, rather than reasoning purely deductively [59]. Hence design evaluation metrics to reflect more rigorous measure accurate defeasible reasoning. In case where model predicts the label correctly, and the label is one of proved disproved (where an actual proof exists), we measure whether the proof generated by the model is correct or not. For this purpose,
compute two automated proof accuracy metries (named Rule FI and Conflict FI) and one manual metric (named Overall Proof Accuracy) as described below. For Rule FI, we extract the rules used in the golden proof and the ones in the proof generated by the model that used to derive new facts (and ultimately, the goal). Then we compute the Fi-score of the overlap of the two sets. For Conflict FI, extract the conflict resolutions (corresponding the gold proof and the ones in the proof generated by the model, and compute the Fi-score of their overlap. For Overall Proof Accuracy, manually verify whether the proof is correct for 50 sampled examples per model. We compute these metrics on depth of the dataset.
pairs of rules) used in
According to the results in Figure 5, all models per- form relatively well in selecting the correct set of rules for the proof. The few-shot model performs poorly conflict resolution whereas the tuned models perform substantially better, suggesting that preference under- standing and conflict resolution do not surface with simple few-shot prompting, and tuning required for models to exhibit this capacity. Second, the models often generate wrong proofs, even when they predict the label correctly. The issue is less severe in the case severe
80
60-
20
accurary
Finetuned pronts
Promohcuannt wi GoT Teanhhe wi OXT
Figure 5: Proof accuracy metries for various models on depth 2 of the dataset, when the label is predicted correctly.
of the prompt-tuned model but becomes for the finetuned and few-shot models. We provide examples of proof failures in Appendix A.
5.3 Do Conflicts Make Reasoning More Difficult?
We create four versions of BoardgameQA named NoConflict, LowConfiict, Medium- Conflict, and HighConflict, with PConf set to 0.0, 0.2, 0.5 and 0.8 respectively: other fac- tors are kept the same. Note that the Medi- umConflict corresponds to the dataset in Fig- ure 4. The results of the models on these datasets reported in Figure 6. The perfor- mance of all models monotonically degrades the number of conflicts increases, show- ing that conflict resolution is indeed major factor in the difficulty of the problems. For example, BERT performs above-random for the NoConflict and LowConflict cases, but the model performance drops to near-random MediumConffict and HighConflict cases.
Majority Claes Conflict Centlct High Contlctt
iBent Larget
113 XXLI
628)
INULM 5408)
Figure 6: The model performances on four versions of the BoardgameQA dataset with various amounts of conflicts them.
5.4 Which Conflict Type is More Difficult Resolve?
To test which type of conflict (See sec. 4) is more difficult for the models, we create three versions of the dataset with varying proportions of Typel vs Type2 conflicts, by setting PCouTypel to 0.2, 0.5, and 0.8 respectively. The first dataset mostly contains conflicts of Type the second contains both

---

conflicts in a similar amount, and the third dataset contains mostly Type2 conflicts. The other factors kept constant across the datasets.
The results the models are reported Fig- ure 7. We that models perform slightly better on the dataset with mostly Typel con- flicts. This discrepancy between performance on Typel and Type2 conflicts is intuitive because in the case of Typel conflicts, the model can ignore the conflicting rule and whether its body can be proved, but in the case of Type2 conflicts, the model has to show that at least one of the elements in the body of the conflicting rule cannot be proved. In the case of tuned models, we furthermore observe that biasing the dataset toward one conflict type results better performance overall. This might be because the model mostly needs to learn to resolve one type of conflict which may be easier than learning both.
Najoriy Cas
Nouly Sppi
Buth types
Musty 1pe2
(BERT Lange)
(TS XX11
[PLLM 620)
(PiLM 54100)
Figure 7: The model performances on three versions of the BoardgameQA dataset with different distribu- tions on the type of conflicts.
5.5 Does Information Incompleteness Make Reasoning More Difficult?
As described Section we can control the amount of information incompleteness using parameter which we named PMisstffo To test how the information incompleteness affects the performance of various models, we create three versions our dataset with PMissifo set 0.2, 0.5 and 0.8, which we name KnowledgeLight, KnowledgeMedium and Knowledgelleavy, respectively.
The results are reported Figure 8. We ob- serve that as the amount of required knowl- edge increases, the performance of the fine- tuned models decreases uccordingly. How- ever, the performance of the prompt-tuned and few-shot models remain relatively un- changed, likely due to the larger size of the
Majrtty Gans
Know edge &. iht
Know edge Metium
Know edge Haavy
model and the extra amount of knowledge that present the model, as well as the fact that working with real-world knowledge might easier for these models than with artificial knowledge.
Rineuueed with proees Riseuseed wi proot. Promptcuned (BEAT Lange) 1T' XX11
Fawarar (Pil 560R0
Figure 8: The model performances on three versions of BoardgameQA with various degrees of incomplete information.
5.6 Do Distractors Make Reasoning More Difficult?
We also measure the effect of distracting facts and rules on model performance. distract- ing fact or rule is one that does not appear in the proof and does not change the label. In Figure 2, for example, "the frog has knife" distracting fact. To this end, each time we call Algorithm 1, besides the sampled sub- questions, we also sample some distracting sub-questions and add them to the set of sub- questions. We create three versions of the BoardgameQA dataset where we add 0, 1, and distracting facts in each step, which we name NoDistractors, SomeDistractors, and ManyDistractors, respectively.
Aretuned weo pridfs Anetuned proofs Promptoueed (X5 628)
Fewshor wi
Figure 9: The model performances three versions of BoardgameQA with various amounts of distracting facts and rules.
According to the results Figure 9, the performance of the tuned models does not substantially degrade with a small number of distractors, potentially because the distractors can help the model avoid learning spurious correlations. However, their performance drops substantially with more dis- tractors. Also, with more distractors, the performance of the few-shot model decreases monotonically,

---

although only marginally (this observation is consistent with the results of [42]). This shows that distractors (that are typically common in real upplications) can also compound the problem difficulty.
Limitations
Our dataset, its current form, focuses primarily on deductive logical entailment, where the problem is a classification problem (label с (proved, disproved, unknoun]) }), and the contradictions are also binary (i.e. one rule suggesting something is True and the other suggesting it is False). Future work can extend BoardgameQA and the analysis provided in this work to non-classification cases where 1- one needs to apply defeasible logical reasoning to answer questions such as "Who will be attacked by the dog?". 2- one needs to resolve non-binary conflicts where, e.g.. one rule suggests "the dog is currently in Canada" and the other suggests "the dog is currently in Australia", 3- there are conflicts and preferences over facts as well, e.g.. Fact1: Fiona has travelled to every country in Europe, Fact2: Fiona has not travelled to the Scandinavian countries, Fact2 is preferred over Facti.
The current work assumes the initial state (facts) and the rules of the game are small enough to be included in the prompt. It is also limited to deductive reasoning with the modus ponens rule. Future work can extend BoardgameQA and our analyses to the cases where not all the facts and rules can be included in the prompt due to the limitation in the prompt length, as well as the case with other types of rules such as proof by contradiction, disjunction elimination, etc (see [42]).
In this work, we only studied one simple but highly practical solution to conflict resolution (i.e. based on preferences). Future work can extend BoardgameQA and the analysis in this paper other natural types of conflict resolution. Note that in some applications, preferences for conflict resolution have to be assigned with great care and diligence to avoid unfair treatment of information sources.
Conclusion
In this work, we introduced BoardgameQA, dataset for measuring the natural language reasoning ability of language models (LMs) in the presence of conflicting input sources. Our dataset furthermore includes scenarios in which the knowledge required for reasoning only partially provided as input and additional information needs to come from the model itself. We tested several types of LMs on different variations the dataset and observed that LMs perform poorly when reasoning with conflict- ing inputs. In the case of smaller models, the performance was also poor when additional knowledge from the LM is needed. Since reasoning over contradicting and incomplete sets of information is a common scenario in real-world applications, our results highlight an important gap in the reasoning capacity of current LMs. We hope our dataset can guide future work developing methodology to improve the reasoning ability of LMs under this setup, finding alternative formulations of conflict resolution that better facilitate LM reasoning.
Acknowledgements
We thank Yue Liu, Tania Bedrax- Weiss, and the anonymous reviewers for their valuable feedback.
References
[1] Allaway, E., Hwang, J. D., Bhagavatula, C., McKeown, K., Downey, D., and Choi, Y. Penguins don't fly: Reasoning about generics through instantiations and exceptions. arXiv preprint arXiv:2205.1 1658, 2022.
[2] Arabshahi, F., Lee, J., Gawarecki, M., Mazaitis, K., Azaria, A., and Mitchell, T. Conversational neuro-symbolic commonsense reasoning. In Proceedingy of the AAAI Conference Artificial Intelligence, volume 35, pp. 4902-4911, 2021.
[3] BehnamGhader, P., Miret, S., and Reddy, S. Can retriever-augmenter language models reason? the blame game between the retriever and the language model. arXiv preprint arXiv:22 12,0022,, 2022.
10

---

[4] Betz, G., Voigt, C., and Richardson, K. Critical thinking for language models. In Proceedings of the 14th International Conference on Computational Semantics (/WCS), pp. 63-75, Groningen. The Netherlands (online), June 2021. Association for Computational Linguistics. URL https: //ac //aclanthology www.y...g. org/ 2021 iwcs-1.7.
[5] Bhagavatula, C., Bras, R. L., Malaviya, C., Sakaguchi, K., Holtzman, A., Rashkin, H., Downey, D., Yih, S. W.-t., and Choi, Y. Abductive commonsense reasoning. arXiv preprint 1rXiv:1988.05739 2019.
[6] Bhakthavatsalam, S., Anastasiades, C., and Clark, P. Genericskh: A knowledge base of generic statements. arXiv preprint arXiv:2005, 00660, 2020.
[7] Billi, M., Calegari, R., Contissa, G., Lagioia, F., Pisano, G., Sartor, G., and Sartor, G. Argumen- tation and defeasible reasoning in the law. J, 4(4):897-914, 2021.
[8] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, D., Dhariwal, P., Neelakantan, A., Shyam, P. Sastry, G., Askell, A., Agarwal, S., Herbert- A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C.,, Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, Language models are few-shot learners. Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL wtt.a/aaaa.....1eeaaaraaaaaaaaa. Paper pdf.
neurips cc/
paper_filest wtt__fflee/ppppeee wwtpss/aapppp/eeeeeeeeeeeeeeeeae
[9] Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du N., Hutchinson, B., Pope, R., Bradbury, J., Austin, J., Isard, M., Gur-Ari, G., Yin, P.. Duke, T., Levskaya, A., Ghemawat, S., Dev, S., Michalewski, H., Garcia, X., Misra, V., Robinson, K., Fedus, L., Zhou, D., Ippolito. D., Luan, D., Lim, H., Zoph, B., Spiridonov, A., Sepassi, R., Dohan, D., Agrawal, S., Omernick, M., Dai, A. M., Pillai, T. S., Pellat, M., Lewkowyc7, A., Moreira, E., Child, R., Polozov, O., Lee, K., Zhou, Z., Wang, X., Saeta, B., Diaz, M., Firat, O., Catasta, M., Wei, J.,
K.,
Eck. D., Dean, J., Petrov, S., and Fiedel, N. PaLM: Scaling language modeling with pathways.
arXiv:2204.00 02311, 2022. https: //axiiv .rg/ orggabs/////ooooooooogooooee 2204. 02311.
[10] Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., De- hghani, M., Brahma, S., et al. Scaling instruction finetuned language models. arXiv preprint arXiv:2210.11 11416. 2022.
restuuctio-finetueed
[11] Clark, P., Tafjord, O., and Richardson, K. Transformers as soft reasoners over language. In Proceedings of the Twenty-Nitth International Joint Conference on Artificial Intelligence, 20CAI'20, 2021. ISBN 9780999241165. URL https: //dl. acm. awm.ara/.......................a /d////i///// aaaaa/aa/0/ 5555/ 3491440 3491977.
[12] Creswell, A., Shanahan, M.. and Higgins, I. Selection-inferenee Exploiting large language mod- els for interpretable logical reasoning. In The Eleventh International Conference on Learning Representations, 2023. URL https::/ /owp://i/ee/eeeeee///eee net. faet/forum? wor.miidddaaa -d- A4.
[13] Dalvi, B., Jansen, P., Tafjord, O., Xie, Z., Smith, H., Pipatanangkura, L., and Clark, P. Ex- plaining answers with entailment trees. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7358-7370, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 1/253/22/2021 emnlp-main.585. URL https:tttcclcttththto/
org/ 2021 emnlp-main. 588.
[14] Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1010.01810.044000 2018
[15] Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., and Berant, J. Did aristotle laptop? question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguisties, 9:346-361, 2021.
[16] Gómez, S. A., Chesnevar, I., and Simari, G. R. Defeasible reasoning in web-based forms through argumentation. International Journal of Information Technology & Decision Making, (01):71-101, 2008
11

---

[17] Guu, K., Lee, K., Tung, Z.. Pasupat. P., and Chang, M. Retrieval augmented language model pre-training. In International conference on machine learning, pp. 3929-3938. PMLR, 2020.
[18] Han, S., Schoelkopf, H., Zhao, Y. Qi, Z., Riddell, M., Benson, L., Sun, L.., Zubova, E., Qiao, Y., Burtell, M., et al. FOLIO: Natural language reasoning with first-order logic. arXiv:2209.008.0 2022. URL https: /ttp/aaxvvv////////////e///aaav. org/ abs//20// 00840..
[19] Hecham, A., Bisquert, P., and Croitoru, M. On a flexible representation for defeasible reasoning variants. In AAMAS 2018-17th International Conference on Autonomous Agents and MultiAgent Systems, number AAMAS' 18, pp. 1123- 1131, 2018.
[20] Hewitt, C. Planner: language for proving theorems in robots. In Proceedings of the 1st International Joint Conference on Artificial Intelligence, IJCAI'69, pp. 295-301, San Francisco, CA, USA, 1969. Morgan Kaufmann Publishers Inc.
[21] Hu, S., Luo, Y., Wang, H., Cheng, X., Liu, Z., and Sun, M. Won't get fooled again: Answering questions with false premises. In ACI, 2023.
[22] Katz, U., Geva, M., and Berant, J. Inferring implicit relations in complex questions with language models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 2-548-2566, 2022.
[23] Kazemi, S. M., Kim, N., Bhatia, D., Xu, X., and Ramachandran, D. Lambada: Backward chaining for automated reasoning in natural language. In ACL., 2023.
[24] Khot, T., Trivedi, H., Finlayson, M., Fu, Y., Richardson, K., Clark, P., and Sabharwal, A. Decomposed prompting: A modular approach for solving complex tasks. In The Eleventh International Conference on Learning Representations, 2023. URL https::/oppereeiier net/forum? id= "=noouunndddnnng2zjjaaa.. _nggzQjzaRy.
[25] Kim, N. and Schuster, S. Entity tracking in language models. arXiv preprint arXiv:3305.02363. 2023.
[26] Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning, arXiv preprini 1rXXi:2104408690 2021.
[27] Madaan, A., Tandon, N., Rajagopal, D., Clark, P., Yang, Y., and Hovy, E. Think about it! improving defeasible reasoning by first modeling the question scenario. arXiv preprint urXiv:2110.12344 2021.
(28) Maher, M. J., Tachmazidis, I., Antoniou, G., Wade, S., and Cheng, L. Rethinking defeasible reasoning: A scalable approach. Theory and Practice of Logic Programming, 20(4):552-586 2020.
[29] McCarthy, J. Programs with common sense. In Proceedings of the Teddington Conference on the Mechanization of Thought Processes, pp. 75-91, London, 1959. Her Majesty's Stationary Office. URL http: /
stanford. edu/ aadu/jmcmmcc59 .tmll.
[30] Min, S., Zettlemoyer, L., Hajishirzi, H., al. Crepe: Open-domain question answering with false presuppositions. In ACL, pp. arXiv-2211, 2023.
[31] Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowyez, A., Bosma, M., Luan, D., Sutton, C., and Odena, A. Show your work: Scratchpads for intermediate computation with language models. In Deep Learning for Code Workshop, net wetrruuiid-Hlix2iieee HB1x2idbkbq.
2022. URL /
[32] Pan, L., Albalak, A., Wang. X., and Wang, W. Y. Logic-Im: Empowering large language models with symbolic solvers for faithful logical reasoning. arXiv preprint arXiv:2305. 12295, 2023.
[33] Pollock, J. L. Defcasible reasoning. Cognitive science. 11(4):481-518, 1987.
[34] Poole, D. A logical framework for default reasoning. Artificial intelligence, 36(1):27-47, 1988.
[35] Qiao, S., Ou, Y., Zhang, N., Chen, X., Yao, Y., Deng, S., Tan, C., Huang, F,, and Chen, H. Reasoning with language model prompting: A survey, arXiv preprint 2x1v:2212.005597, 2022.
12

---

[36] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-555) 2020.
[37] Reiter, R. Nonmonotonic reasoning. In Exploring artificial intelligence, pp. 439 481. Elsevier, 1988.
[38] Roberts, A., Chung, H. W. Levskaya, A., Mishra, G., Bradbury, J., Andor, D., Narang, S., Lester, B., Gaffney, C., Mohiuddin, A., Hawthorne, C., Lewkowycz, A., Salcianu, A., van Zee, M., Austin, J., Goodman, S., Soares, L. B., Hu, H., Tsvyashchenko, S., Chowdhery, A., Bastings, J., Bulian, J. Garcia, X., Ni, J.. Chen, A., Kenealy, K., Clark, J. H., Lee, S., Garrette, D.. L..LeeTorrp. J., Raffel, C., Shazeer, N., Ritter, M., Bosma, M., Passos, A., Maitin-Shepant J., Fiedel, N., Omernick, M., Saeta, B., Sepassi, R., Spiridonov, A., Newlan, J., and Gesmundo, A. Scaling up models and data with t5x and seqio. arXiv preprint arXiv:2203. /7189, 2022.
URL httph:ttp////tctt /tsttps://rxiiis //wpxv/a.vvvvv/aa//////eaaaaaaa http//iwwivv......aa........ org/ haabs/220... 17189.
[39] Rudinger, R., Shwartz, V., Hwang. J. D., Bhagavatula, C., Forbes, M., Le Bras, R., Smith, A., and Choi, Y. Thinking like skeptie: Defeasible inference in natural language. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4661-4675, 2020.
[40] Saeed, M., Ahmadi, N., Nakov, P., and Papotti. P. RuleBERT: Teaching soft rules to pre-traie pre-trained language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pp. 1460-1476, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 118653/v1/2222 2ttp://.112211..1-mmmeeeee. wewnlp-main.n.... 10. URL http://///aaaaahhlooossee org/2021 emnlp-main 110.
[41] Saparov, A. and He, H. Language models greedy reasoners: A systematic formal analysis of thain-of-thought. In The Eleventh International Conference on Learning Representations, 2023. aett//oorumm.ddqFFBBBa.... URL https: http////ppee/eeeeeeeeeeeeeeeeeee
[42] Saparov, A., Yuanzhe Pang, R., Padmakumar, V., Joshi, N., Kazemi, S. M., Kim, N., and He, H. Testing the general deductive reasoning capacity of large language models using ood examples. arxiv preprint arXiv:2305.15266 2023.
[43] Sartor, G. Defeasibility legal reasoning. Springer, 1995.
[44] Shoenfield, J. Mathematical Logic. Taylor & Francis, 2001. ISBN 978156881 1352. URL https://books: http://oooos.ooggeee... www..///oooiiieeeoccoooo htt.//ooooosii////AAAAAAA.
[45] Sinha, K., Sodhani, S., Dong, J., Pineau, J., and Hamilton, W. L. Clutrr: A diagnostic benchmark for inductive reasoning from text. arXiv preprint /908.060 2019.
[46] Sprague, Z., Bostrom, K., Chaudhuri. S., and Durrett, G. Natural language deduction with incomplete information. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 8230-8258, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https: pettps:/acclataaie //////aaaaaaatooyyyy.a hwwpcchooog..rrg//ee org/2022. emnlp-mate main. 564.
[47] Sun, H., Cohen, W. W. and Salakhutdinov, R. Conditionalqa: A complex reading comprehen- sion dataset with conditional answers. arXiv preprint arXiv:2110.06884, 2021.
[48] Tafjord, O., Dalvi, B., and Clark, P. ProofWriter: Generating implications, proofs, and abductive statements over natural language. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 3621-3634, Online, August 2021. Association for Computational Linguistics. doi: 18.18653/v ww2.222...ninnggaaa..333 URL https: httpps//aaclaattooolgg /accaaaanatttoyy...... org/
2021. findingnnnninn acl 317.
[49] Talmor, A., Herzig, J., Lourie, N., and Berant, Commonsensequ: question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.000 2018
[50] Talmor, A., Tafjord, O., Clark, P., Goldberg, Y., and Berant, J. Leap-of-thought Teaching pre-trained models to systematically reason over implicit knowledge. Advances in Neural Information Processing Systems, 33:20227-20237, 2020.
13

---

[51] Wang, B., Deng, X., and Sun, H. Iteratively prompt pre-trained language models for chain of thought. In Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pp. 2714-2730, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https::/acllatthol..gg
org /2022.2 emnle-mallet maip.. 174.
[52] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., and Zhou, D. Chain-of- Chain-of-thought prompting elicits reasoning in large language models. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 24824-24837. Curran Associates, Inc., 2022.
[53] Weston, J., Bordes, A., Chopra, S., Rush, A. M., Van Merriënboer, B., Joulin, A., and Mikolov, T. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv: 1502.05698, 2015.
[54] Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305. 10601, 2023.
[55] Ye, A., Cui, C., Shi, T., and Riedl, M. O. Neural story planning. arXiv preprint arXiv:2212.08718, 2022.
[56] Yu, F., Zhang. H., and Wang, B. Nature language reasoning, survey. arXiv preprint arXiv:2303. 14725, 2023.
[57] Yuan, Q., Kazemi, M., Xu, X., Noble, I., Imbrasaite, V., and Ramachandran, D. Tasklama: Probing the complex task understanding of language models. arXiv preprint arX.:22308.15299 2023.
[58] Zelikman, E., Wu, Y., Mu, and Goodman, N. STaR: Bootstrapping reasoning with reasoning. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 15476-15488. Curran Associates, Inc., 2022.
[59] Zhang, H., Li, L. H., Meng. T., Chang, K.-.,, and Broeck, G. V. On the paradox of learning to reason from data. arXiv:2205. 11500, 2022. URL https: / /arxiv..vvv org/ ha//a////// /200/2205...m 11502.
[60] Zhang, H., Li, Z., Huang, J., Naik, M., and Xing, E. Improved logical reasoning of language models via differentiable symbolic programming. In First Workshop on Pre- training: Perspectives, Pitfalls, and Puths Forward at ICML 2022, 2022. URL https: / //openreview net
[61] Zhong, W., Wang, S., Tang. D., Xu, Z., Guo, D., Wang, J., Yin, J., Zhou, M., and Duan, N. AR-LSAT: Investigating analytical reasoning of text. arXiv preprint arXiv:2104.066 06598, 2021.
14

---

Majorty Class
Depth1
20-
Depth?
Dupth33
0-
finetuned (TS XXLI
CoT (PaLM 540B)
(BERT Large)
(PiLM 628)
Figure 10: The model performances various depths of binary version of the BoardgameQA dataset.
Depth Etxample: lew players am playing a boardgame, The curment state the game as follwos has dollars dregan lenses the houses nocupied by the liger shoek has dollers. campine cumently Cape
rules the game are fnilows Regaeding vampin, more money then the There ersos enimal then we can hiuusss occappedd by lige??
shirk and bew contined then wa concllde diarms bulking
vampir deftniely
preferned buildog. Regarding vampire, It
ditaam buildag, bulldoo??
preterred
game and niles preterences,
vamp
Model Output: the nampine has more money
and the combinees sed and acconding to Rulet if the vampiie hiss more manay shark we cannot prove antecedent there usista animal that are occu plad by Iger vampre cisamma bul dog. So statement vampire disams bulldog is proved
combinsd than we can concludu that it disams the bulldag and for conficting and priority
"yes".
Error Analysis:
Rutell model that model belleves Rule! activates, should money
moneay
that that Ruin2 does
that exists aninal which
the
the based racts.
Figure 11: Highlighting some of the model errors for depth I example when using PaLM 540B.
A More Experimental Results and Analysis
Binary Classification: Kazemi et al. [23] observed that reasoning with unknown labels is particularly challenging for few-shot LMs, because providing a natural chain-of-thought for unknown is difficult. To measure if the poor performance is merely due to the existence of examples with unknown label or due conflict resolution being difficult for these models, we also created a binary version of the dataset for depths 1, 2. and 3 where only examples with proved and disproved labels are included." The results are reported in Figure 10. We overall see similar patterns as the binary case, except for some improvements for the T5 model on depth 2.
Examples of Model Failures: To showcase some of the reasons why models struggle with the Boardgame dataset, in Figure 11 13 we show some example proofs where the model made variaty of errors and Figures 15-24 we provide specific examples where the model generated wrong proofs. Some of the dominant error cases (showcased in the examples) include: 1- hallucinating or misunderstanding conflicts and preferences, not being able correctly fill in the incomplete information, 3- misunderstanding logical rules, 4- failing to prove both elements in a conjunction, 5- getting distracted by distracting facts and rules and going on wrong proof path, and 6- being unfaithful to the provided facts and rules and changing them so proof can be found in the case where no proof exists.
The confusion matrices for the model predictions with respect to the golden labels on the BoardgameQA dataset is provided in Figure 14 for various depths. One interesting observation is that for models tuned with proofs, while the model perform well at predicting unknown labels in lower depths, in higher depths they tend generate proofs (with proved or disproved labels) even when the label is unknown (e.g., for examples with label unknown in depth the prompt-tune model predicts unknown for 268 instances, while depth 2 predicts unknown only for 2 instances). This may be because when the search space for proof increases, LMs cannot verify all possible solutions and decide that the label unknown. Instead, they start a path in the hopes that ends in a proof.
Confusion Matrices:
Standard Deviations: Due to the high cost of experimenting with large language models, the results in the paper are reported based on a single run. To gain insight into the standard deviation of the
'In this case, we set Pcon/ 1.0 for the first call we make to Algorithm 1: Otherwise, the dataset will have spurious correlation that can be exploited without doing any reasoning.
15

---

Depth 2 Example: tow players are playing boandgame The current state the game follcws, The gorilo is warching a movie from 1982. The woodpecker has foottall with maduss ot 20 inches, and sweare to the duck The zebm netme in the poot next to the house the bu butterfty And the rules the game are as tollowe Rule!: The gorill will not enjy the company vy of the va sampre Ifit (th gorill) works in healthare. Ruke2: The goriiia encyys the company bny the vampre whenever at kess one animal avims insice the pool located besides the house the butterfly Rules: there is ovidence that one anima, no mator which one, coquires a phosograph of the peatow, thon the woodbecker calls the vampire undou boodly. Ruled; Here is an important ploce of intormation about the wooopecker. I has football that fts an 58.4 61 x 59.6 inches box then does not co the vampiee for sure. Ruiss: at least one anima leaves the houses that are cccupted by the dragon, then the vomple does important place informaiio about gorills
dieam bullikog. Ruieé: Here
walching movie was reloaased
the frus man landed on moon socupled the dragen, loo.
vamssru sure. Rule?: samehhng swears to the duck, then leaves the houses Rulet proformed over Rulo2. Rukc is peeforred over Rules, Ruled profiomed over Rule2 Based on the game state and the rules and profer onces, does the vamotro bulldog?
then it doea not unjoy the companiorahhie
Model Cutput: Sirce the woodpecker has a footrell with a medius inches and acconting to Ruted the unodpecker has foothel that titx a 58.4 x dS inches then it down not call the vemrtire we wondowcker dives not cul vempine Sinoe woodlpeccter dows not call cumpine wozzrdinng Ruukd thene accurees photograph the poatow then the woccpedier calls vampire so the woocpedker does not acquire the weadpeceer does not acquire photograch the peatowl according Ruedd pccupied dragon then the va vampre does diearm buildog the vampre dicorms the bulldogg So the statement the vampie disarms the bulldag proved and answer is "yee".
evidence that one animal, moster
photograph the seatow,
one animal leaves housed ohe
Error Analysis:
Since Ruted actvvaes, model ossumes the woodpocker Rute3 does activate
the vamphr without checking whother Rutci (the higher prionity activates
Lucklyy the model
While Runlll indledd activatas the model makes a planning untrr by starting from Ruied. The correct preaf plan to start with Rule7 fallowed RuleS.
An insiially wrong proof plan leads conseiquent wrong stops in the proof planning (Ru is followed by Rute8).
The model misundersten nnuunderstadd the antacedent and the consequont of Rules.
The model jumps Rures and antiee a condusion about the question, without proxing the ansecedent of Rulss. The modil min. www.nneeeneeddee the sign of the consequent of Ruleé.
Figure 12: Highlighting some of the model errors for a depth 2 example when using PaI.M 540B.
Depth 3 Example: A few players are playing a boardgame, The cumeen state of the game as follows, The buttertly swears to the wall The cougor has mends, and is namedl Coaccpenn The crow is named Charte The dove has a beokwfbst with a diameter inches, and is named Texza The dove has card is whit color The monkey has card that wiite color. The moneyy named Mojo shark is named Poso. worn named Tango The husky does
down beedo. Rule2: trog retuses to dbserving an animal brings an tank for bison, one can conclude the dalmatian going to invest company owned by animals manages convince cstrch dalmastan, tame time smiles beetl?? Then you can does cortain that Nuthorlands,
caste beongs to the finshh
rures
game are follow,, Rulet:
tollowing that animal does not smile
beetle, Ru Are you contain
nigodiates
badger.
appears
pelkan unquesionably lotior the shark's has information about
the paliiar, leterr
from
Rule8: the
hide
from the polkknn Rulo7:
trrre important
i'lt hos wiizee find latler the seme the find lelter of the warmbe meme, tran
letter
the firs
crow's then manages conwirnce
the owtrich for Ruted: Raggnnding the dove, illl wogs dial with the badger and the palliinn ahouts at the badger, then the bedger falls square of the companioonal loraiip of the delmutiann Rutdl: the
conclude thit wejoyys
mouse. Ru One the rulea the game that if the dove enjoy the companionalle up of the damallan, then the calmatian will without hesitation invest in the company whose cener ls the beeds. Rule11: The cougar will smile at the bodte Ifil the oougar) has more than soven friends. Rule12: This is basic rule: the finch negolwiee a deal with the pellees then the conce usion thet the ow loon will not shout at the bedger lol mos immrdcstelly and effectieel Rude This a bosic new if the
busky dows not teer down the padta that bwiking the frnch then the concuusion that the finh negedistes dieal with the palktan follows immediatllyy and wllectioyy.
Ruld is prefemee over Rull11 Rule2 is profaried over Rusr00 Rules is prefermed over Rule12 Based on the game state and the nires and prefee fall on aquare of the mouse?
doss the badlger
Model Output: Since the cougar has 8 thenns and according ng to Rulett it cougar has more than soven thiends than the cougar will smile at boote so the cougar smiles at the beetie Since the couger omile lee the beetle and according to Rulee it the cougar smiles the beetie then the cougar does not negotane deal with the
bedger so the cougar does not nogudiate a deal with the badger. Bince the cougar does not negodiate a deal with the badger and according to Rubid the cougar negotiates a doal with the badger and the palikan shouts at the badger then the badger fails on squaro of the mouse so the badger coos not fall on square of the mouse. 3o the statement the badger falls on a square of the mouse is disproved and the answer is "no".
Error Analysia:
Sinca Ruluf1 activatten, the modul assures es the cougar will ancie the busili without chacking wi whee Ruwd (the hightr priority rule) activates not. Luckky
the model, though, Rule1 does activale.
The model halluchates non exising g instead Ruic2 wi ich makes shortout to conclusion. conumision sign consequent
by only ig the
model
Figure 13: Highlighting some of the model errors for a depth 3 example when using PaL.M 540B.
results, we trained the BERT model (our cheapest baseline) 10 times on each of the datasets from the main text and computed the standard deviations. The standard deviations ranged from 0.3 to 2.6.
B Experimental Details
We conducted our experiments on v3 TPUs for all the models, except for the 540B models where we used v4 TPUs due to their larger size. All the experiments were done using the T5X framework [38] available at https: : httpt/gttttt.t/////eeeeeeeee.ett ctw gooogee
http:/ooooeeerrercc/ettteeeeee research/t5x.
For the fewshot experiments, to ensure the demonstrations and the question fit in the prompt, due to the large size of the examples in BoardgameQA. we only included one example per label as demonstration (i.e. 3 examples one with label proved, one with disproved, and one with unknown in the case of 3-way classification datasets). For each example in the test set, we selected the demonstrations randomly from the training set, while ensuring that they contain both types of conflict resolution. For the prompt-tuning experiments, we used prompt-size of 100 in all experiments, as it worked best in the experiments of [26]. We allowed a maximum of 50K training steps with a batch t
size of 8, a learning rate of 0.1, and weight decay rate of 0.0001. We evaluated the model after each 1000 steps on the validation set and selected the checkpoint with the best validation accuracy for testing. For the finetuning experiments, we also set the batch size to 8 but set the learning rate to
16

---

Depth1
Depth2
Depth3
proved
Thue tabel
Thue lbbd
disproved
disprovid
disproved
unkooow
154
unkrrown
144
105
unknown
64
1600
105
proved
disproved unknoww Predicted label
proved
disrrvved unknown Predicted label
proved
disproved unknown
Predicted label
(a) Finetuned w/o proof' (BERT Large)
(b) Finetuned w/o proofs (BERT Large)
(c) Finetuned w/o proofs (BERT Large)
Depth1
Depth2
Depth3
proved 293
19
proved
230
52
52
proved
222
58
Tha label
The
disproved
5
311
17
disproved
16
299
18
Thue
disproved
16
2.90
27
untroww
12
318
unknown
145
22
166
unknown-
196
49
88
proved
disprowed urknown Predicted label
proved
disproved unknown Predicted label
proved
disproved unknown Predicted label
(d) Finetuned w/ proofs (T5 XXL)
(e) Finetuned w/ proofs (TS XXL)
(I) Finetuned w/ proofs (TS XXL)
Depth1
Depth2
Depth3
proved
295
16
provod
186
17
131
proved
284
45
Thus lakel
disproved
24
2.93
disproved
292
26
True
disproved-
24
3,2
unknown
62
268
unknown
122
15
196
unknown
304
27
proved
disproved Predicted label
unknown
proved
disprooved unknown Predictedd label
proved
disproved unknown Predicted label
(g) Prompt-tuned w/ CoT (PaLM
(h) Prompt-tune tundd w/ CoT (PaLM 62B)
(i) Prompt-tume w/ CoT (PaLM
62B)
62B)
Depth1
Depth2
Depth3
proved
240
48
46
proved
243
40
51
proved
216
33
85
Thee Gibal
elsproved
35
248
50
disproved
39
238
56
disproved
56
1882
95
unitoown
188
96
49
unknown
215
58
60
unkonwwn
21.4
32
87
disproved Predicted unknown
proved
disproved
unknown
proved
disproved unknown
Predicted
Predicted label
ii Fewshot w/ CoT (PaLM 540B)
(k) Fewshot w/ CoT (PaLM
(1) Fewshot w/ CoT (PaLM 540B)
S40B)
Figure 14: Confusion matrices for various models on the BoardgameQA dataset.
0.001 and allowed 50K epochs (since required more time converge) and reported the results for the epoch that achieved best validation accuracy.
17

---

The seahorse has a baskelbal with a diameter 27 inches, and currenily in Istanbuu.. The woll does not lake over the emperor of the snake. And the rules of the game follows.
Facts:
Rules: [..] Rule2: the seahorse has basketball that fits in 37.1 37 x 23,9 inches box, then the seahorse surrenders to the surrender the if (the seahorse) is Turkey at the moment. [...]
swan. Rule3:
Preferences: Rule5 preferred over Rule4.
Question: Does the swan swear to the wandpecker?
Istanbul and according Rule3 the seahorse Turkey surrenders to the swan so the seahorse surranders to the swan. Since the seahorse surenddrrs to the swan and according to Rule2 fits
Prediction: Since
seahorse
37.4 23.9 has basketball that fits in 37 x 37.4 23.9 inches box. [..]
seahorse
then the surrenders
seahorse
Figure 15: An example of a wrong proof generated by PaLM 540B (fewshot) where the error is due to misunderstanding logical rule (given fact f and a rule f' + f the model concludes that f must be true.
fish. [-.]
Rules: [..] Rule2: There exists animal which dances with the fish? Then, the dachshund definilely does not capture the the most important piece) of the owl. Rule3: something does not capture the king (l.e, the most important piece) the owl and additionall not disarm the zebra, then disarms the woodpecker, [-.]]
Preferences: Rule1 is preferred over Rule5. Rule3 is preferred over Rulo4.
Question:
woodpecker?
Prediction: [..] and according Rule4 [..] and for the conflicting and higher prioriyy rule Rule3 cannot the antecedent "the dachshund does not capture king ow!", [-]]
Figure 16: An example of wrong proof generated by PaLM 62B (prompt- (unpt--tuedd where the error due assuming an antecedent high priority rule cannot be proved, whereas it can indeed be proved.
C
BoardgameQA Details
Here, we provide more in depth details about the generation and properties the BoardgameQAA dataset. A sample of theory and question generation from Algorithm is provided in Figure 25.
C.1 Consistency of the Dataset
A defeasible theory is called consistent if whenever conflict arises, the preferences can be used to resolve the conflict. In BoardgameQA, we aim to produce consistent theories. To avoid inconsistencies and loops, each time we call the function in Algorithm 1, we only allow it to sample from the entities that have not been used other rules and (sub-)questions As an example, if we have a rule such as A c, then when call the algorithm for and b recursively, facts and rules produced for the sub-branch of a and for the sub-bran sub-branch of This way, we ensure that when derive new facts in the sub-branch of a, it does not defeat some of the derivations in the sub-branch of (and the derivations in the later stages of the proof do not defeat the earlier derivations). That is because the the of facts used in the rule bodies are for separate entities, and are therefore disjoint. We also apply defeasible reasoning on the final logical theory to ensure that the question be derived from the theory.
separate entities in the
C.2 Incomplete Information
Here, provide more information about the nature and type of incomplete information in BoardgameQA.
Age: We first generate a positive integer corresponding to the age of one of the players expressed in days. Then we decide if we want to use more than or less than relationship. In the case of the former, we next generate another integer y < x and in the case of the latter y > z. Then, we randomly decide a target unit (days, weeks, months, years) for each integer and convert them to that unit. Let x' and y/ represent
obtained values measured with the new units. Then we add fact [PLAYER] is x' [unit] old and a rule if the [PLAYER] is [more than/less than] y [unit] old then ... The model has to be able to convert units of time and then compare them.
18

---

Facts: The mule dances with the fish. [-.]
Rules: [..] Rule2: There exists an animal which dances with the fish? Then, the dachshund definitely does not capture the king (ie. the most important piece) the owl. Rule3: something does not capture king (l.e. important piece) owl and additionally not disarm the zebra, then disarms the woodpacker. [-.]
Preferences: Rule1 is preferred over Rule5. Rule3 is preferred over Rule4.
Question: Does the dachshund disarm the woodpecker?
Prediction: [..] and according to Rule4 [..] and for the confliciig and higher priority rule Rule3 we cannot prove the antecedent "the dachshund does capture the king the awl". [-]]
Figure 17: An example of a wrong proof generated by PaLM 62B (prompt-tuned) where the error is duc to filling the missing information incorrectly.
Facts: The cougar manages convince the ostrich, and smiles al the beetle. The dalmatian named Chickpea. The pelikan has
friends. The seal is named Charlie.
Rules: Rulef: something smiles the beetle and manages to convince the ostrich, then negotiates a deal with the badger.
Rule2: the dalmatian has name whose first letter the same as the first letter the seal's name, then the dalmatian invests in the company whose
is the beelle. the fallowing: that animal does not shout at the badger. Rule5: If the cougar negoliates a doal with the badger and the palikan shouts the badger, then the badger falls square the mouse.
Rule4: From observing that animal brings an ail tank for the shark,
conclude
Preferences: Rule4 is preferred over Rule3.
Question: Does the badger on square the mouse?
Prediction: Since the cougar manages to convince the ostrich and smiles at the beetle and according to Rule [..] so the cougar negotistes a deal with the badger. Since the couger negotlates deal with badger and according to Rule5 the cougar
negotiates deal with the badger and the pelikan shouts the badger then the badger falls on square of the mouse and Rules is a square mouse. So the statement the badger
preferred over the conflicting and lower priority Rule4 falls on a square of the mouse is proved and the answer is "yes".
badger
An example of wrong proof (but correct label) generated by PaLM 540B (fewshot) where the error is due to 1- failing to prove one element of the conjunction and also identifying a non-existence conflict between two rules.
Figure 18:
Affordance: We manually wrote object propertieslaffordances, and list of items that have those properties. Fxamples of properties include sharp, drink, and music and examples of objects for each of these properties include knife, cappuccino, and flute respectively. The facts are of the form The (PLAYER] has [OBJECT] and the rules are of the form If the [PLAYER] has [AN OBJECT WITH PROPERTY] then ... The model has to know about the object properties to connect the facts and the rules.
We first identify groups of colors based on some property. This includes being primary color, being rainbow color, and being a color in the flag of country X. Then, we generate facts of the form The [PLAYER] has a card which [COLOR] color. and rules of the form If the [PLAYER] has a card whose color is [COLOR PROPERTY] then The model has to have information about colors to connect the facts and the rules.
Colors:
We first generate a positive integer Σ corresponding to the amount of money a player has, then we randomly decide if we want the comparison to be between or three players. We also decide we want to use a more than or less than relation. the comparison is between two players and more than is used, then we generate another integer y < x and if less than is used then >> æ;; if the comparison is between three players and more than is used then we generate y, such that y + x. and if less than is used then y = > x. We then generate facts of the form The (PLAYER i] has x dollars. and rules of the form If [PLAYER i] has [more than/less than] than [PLAYER j] and [PLAYER k] combined, then The model has to do a summation and decide which quantity is more or less.
Money:
Textual Entailment: We manually write multiple pairs of sentences where one implics the other. Examples include (assassinated the mayor, killed the mayor), (struggles to find food, has difficulty to find food), and (purchased a luxury aircraft, owns luxury aircraft). The first element of the pair is used in the fact and the second element in the body of a rule. The model has to identify the entailment.
We manually write a list of cities and the countries they are located in. The city names are used in the facts (The [PLAYER] is in [CITY] right now.) and the countries are used in the rule bodies (If the [PLAYER] in [COUNTRY] right now, then ... The model has to know which city is in which country.
Places:
19

---

Facts: The monkey has card that is orange in color. The monkey has a football with a radius of 29 inches, and is a farm worker.
Rules: [..] Rule4: monkey has baskstball that 26.3 6 24.2 inches box, then the monkey swims in the pool next to the house of the crab. [..]
Preferences: Rule5 is preferred over Rule1.
Question: Does monkey pay money rhino?
Prediction: Since the monkey has a football with radius of 29 inches and according to Rule4 if the monkey has a basketball that fits 25.8 inches
mankey swims the pool to house
[--]
Figure 19: example of a wrong proof generated by PaL.M 540B (fewshot) where the error is due to 1- mistaking a football with basketball, and 2- not being able to fill in the incomplete information by realizing that a ball with a radius of 29 inches does not fit in a 26.3 x 25.6 x 24.2 inches box.
Facts: The gorilla watching a movie from 1952. The woodpecker has football with redius of inches, and sweers to the duck. The zebra swims in the pool next the house of the butterfly.
Rules: Rule1: The gorilla will not enjoy the company of the vampire if (the gorilla) works in healthcare. [...] Rule4: Here an important piece of information about the woodpecker: it has football that fits in 58.4 x 61.5 59.6 inches box then does not call the vampire for sure. Rule5: If al least one animaal leaves the houses that are occupied by the dragon, then the vampire does nol disam the bulldog. Rule7: something sweers the duck, then leaves the houses that are occupied by the dragon, too.
Preferences: Rule1 is preferred over Rule2. Rule3 is preferred over Rule4. RuleB is preferred over Rule2.
Question: Does the vampire disar buildog?
Prediction: Since the woodpecker has football with radius of 28 inches and according to Rule4 if the woodpecker has football vampire woodpecker
that 58.4 59.6 inches box does
call vampire. [...]
Figure 20: An example of a wrong proof generated by PaLM 540B (fewshot) where the error is due to starting with distracting fact that took the proof on a wrong path (the correct proof is to first use the fact The woodpecker swears to the duck and Rule7 to conclude that The woodpecker leaves the houses occupied by the dragon, and then use Rule5 conclude that The vampire does not disarm the bulldog.
Names: We assign name (from list of manually written names) to two of the players and then write rules in the form of If (PLAYER has name that starts with the same letter as [PLAYER 21, then....
We mamually write list of pairs of jobs and the industry they belong to. Examples include (nurse, healthcare), (high school teacher, education), and (sales manager, marketing). We use the job in the fact and the industry in the rule body. The model has know which job is part of which industry.
Jobs:
The facts mention that one of the players has an object (a notebook or ball) and give the dimensions of the object (the height and width for notebook and the radius or diameter for the ball). The rule body asks whether the object fit in box with some given dimensions. The model has to understand how 3D objects fit inside each other to able to connect the fact to the rule.
Volume:
Events: We manually write list of world events and the year when they occurred. Examples include (world war 1 started, 1914), (the first man landed on moon, 1969) and (Obama's 's presidency started, 2009). Then we write facts of the form The [PLAYER] is watching movie from [YEAR] and rules of the form If the [PLAYER] is watching a movie that was released [before/after] [EVENT), then .... The model has to know the time for major world events to be able to connect the fact and the rule.
Friends: We first generate a positive integer corresponding to the number of friends player has. Then, we either generate a fact such as The [PLAYER] has x friends or The [PLAYER] has 21 friends that are [ADJECTIVE] and x2 that are not where T1 + I22 Then we decide if we want to use a more than or less than relation. In the former case, we generate number y and in the latter case y > T. Then generate rule of the form If the [PLAYER] has [more than/less than) y friends, then
Due to the nature of the extra knowledge and reasoning cases we consider, we only add such cases at the last theory generation step of Algorithm I (i.c. when d = 1); Otherwise, will need to follow
20

---

Facts: The reindeer has three friends that are energelic and friend that is not. The reindeer will lurn 18 months old in few minutes, finch
power plant
green
reindeer.
Rules:
Rule1: For the reindeer, the bellef is that the german shepherd disarms the reindeer and the finch builds power plant near to your
reindeer, you can conclusions, Rule2: The leopard trades one of its pieces with the cobra whenever at least one animal builds power plant close to green
green
going acquire photograph
Rule3: Regarding reindeer, photo of the finch. Rule4: Here is important piece of information about the reindeer: if has fewer than 5 friends then acquires photo finch
years
that acquires
Proferences: Rule3 is preferred over Rule1. Rule4 proferred over Rule1.
Question: Does the leopard trade one its pieces with the cobra?
Prediction: Since and according Rule1 and Rule is preferred Rule3 reindeer not gaing acquire photograph of the finch. [..]
Figure 21: An example of a wrong proof generated by PaLM 540B (fewshot) where the error is due to misunderstanding preference.
Facts: The liger has a card that violet in color. The worm invests in the company whose owner is the mouse. The goat does not castle for the pigeon.
Rules: Rule3: The living creature that does not create castle for the pigeon will never hug the liger. [--]
Preferences: Rule2 is preferned over Rule1. Rule6 is preferred over Rule5.
Question: Does the liger invest the compeny whose owner the mermaid?
Prediction: [..]] know goat does not hug the liger"
hug the liger, according Rule3 the goat does not hug liger, then
Figure 22: An example of a wrong proof generated by where the error is due to hallucinating facts and rules.
generating sub-questions and rules for questions such as The dog is named Paco leading to unnatural rules such "If.. ... then the dog named Paco.
C.3 Entities, Predicates, and Templates
Table presents the set of entities, predicates, and templates used BoardgameQA. To make the problem slightly more challenging in terms of language complexity, use different entities, predicates and templates in the test set.
C.4
Sample Proofs
In Figure 26, provide some sample proofs with depths 1, 2, and from the dataset. It can be observed that as the depth of the dataset increases, the length and difficulty the proof increases substantially.
21

---

The llama creates one castle for the camel. The liama has card that white in color. The monkey public relations specialist.
Facts:
canclude that does not suspect the truthfulness of the mermaid. Rule3: If the monkey manages to convince the mermaid, then the mermald is not going to pay money the seahorse. Rule4: For the mermaid, if the belief that the dinosaur dances with mermaid and llama does suspect the truthfulness of the mermald, then you can add "the mermaid pays some $S$ to the seahorse" to your conclusions. Rule5: the monkey works marketing, then the monkey manages convince the mermaid.
Rules: [..] Rule2: Regarding the liama, has card whose color appears the flag Italy, then
Preferences: Rule1 preferred Rule2. Rule4 preferred Rule3.
Question: Does the mermald pay money to the seahorse?
Prediction: know llama has that is in color, appears in the flag of Italy, according to Rule2 [..]
Figure 23: An example of a wrong proof generated by T5 where the the model got distracted and ended up wrong proof path.
Facts:
The otter has a 10 inches notebook. The otter is 15 months old. The duck does not tear down the castle that belongs to otter.
Rules: Rule1: If the otter does not enjoy the companionship of the dugong, then the dugong captures the king (ice. the most important piece) the Rule2: Regarding the oller, it to see the dugong. Rule3: Regarding the otter, if has a notebook that fits in à 18.1 x 12.8 inches box, then we can condlude that it the dugong.
years old, we conclude
want
Proferences: None
Question: Does the dugong capture the king of the bear?
Prediction: Since is months old and 15 months than years and according Rule2 the otter less than years old then does not want to see the dugong so the olter does not want to see the dugong. [...]
An example of wrong proof generated by PaLM 540B (fewshot) where the error is due to changing a rule such a way that a proof can be found, when a proof does not exist.
Figure 24:
(dog. unite, (ion)
Recursive Call (dog. unite lion)
(dag. unite, lion)
Add to facts
respportt cat)
Initill Question (dog. attack, cat) D=1
Sample facts and rule
Sample facts and rule
Canf Type
Cont
Type2
R2: respect, - liclog atbock, cat)
Yes
lion) = (X affack cat)
Add to cules
Add to rules
Adi to Preferenass
R2 Rt
Figure 25: sample of theory and question generation from Algorithm 1. Initially, the question has been selected to be (dog, attack, cat). The input depth D = 1 indicates that theory with one hop of reasoning should be generated. Then a fact (dog, unite, lion) and rule Rl: (X, unite, lion) + (X, attack, cat) have been generated. Notice that the combination of the fact and the rule conclude (dog, attack, cat). Next we randomly decide if a conflict should be generated. The decision is yes, so we generate another fact (dog. respect, car) and rule (dog, respect, cat) !(dog, attack, car). Notice that the two rules have contradictory conclusions now. We next decide randomly the type the conflict, and Type2 is selected in this case. Therefore, facts generated for the conflicting rule, and make recursive calls for the remaining facts, which is only (dog, unite, lion). This call is made with D 0, therefore the stopping criterion triggers and add (dog, unite, lion) to our set facts.
add R2 RI to our preferences, remove one of the
22

---

Sample Depth Proof: We know the dog card that white in color, white starts with "w", and according to Rule2 the dog a card whoce
leller "w", lhen dog preperes armor for eel", and has hightrr preference than the conflicting rules (Ruie1), so we can conclude "hh dog prepares armor for the cat" So the statement "the dog prepares armor for the eef is proved and the answer is 'yee".
the koalo, and according to Rule2 something eats the food the koala, than ralls the dice for the lnopard", so we can conclude "the whale rolls the dice for the loapand". We know the buffalo does not know the defenstve plans of the cal, and according to Rule! "i something does not know the defensive plans of is leopard", fields whose bulfslo does not ellock green
Sample Depth Proof: We know whale eats the
the cal, Ihen doesn't altack leapard". green field whose the whale rolls the dice
condude "the buffalo does not atlack the
leapard loopard, and according to Ruled "If the buflala does attack the green fiokts whose owner the leupard but the whsle rolls the dicee for the leopard, then the leopard raises peace fleg for the zender", and for the confficting and higher priority rule Rule3 we cennol prove the entecadent "the leopard does respect the kudu/, we can peace flag the zander" the statement "the laopard
owner
conclude "the loopard proved and the answer is "yes".
peace flag for zander"
Sample Depth Proof: We know the crocedile is nemed Milo and the donkey nemedd Meadoww bolh namres start with 'M": and according Rule'1 the procodile sleals five paints from the carkroach", and for the conflicting and highar priority rule Rute4 cannot prove antacodant 'at least one animal knows the dafonsive plans of the buffala", so we can concludo "tho crocodie steals tive points from the cockroach" We know the crocddle steals five points from the cockmacch and according Rule6 "If something steels of
crocadle has
whose first
first letter the donkeyys name,
five points from cockroach, crocodie does not remove from board one of pieces of liger.. We know aardvork proceads to the spot right after the lon, and according to Rules "If something proceeds to the spot right after the lon, then it does not eot the food of the spider), and for the conficting and higher priority rule Rule3 we cannot prove the antevedent "at least one animal attacks the green fields whose hold same number of points the phoenix, and according to Rulel something does hold same
does not
from hoard
tiger"
the callish",
we conduue "the sardvark does not eel the food the spider". We know the
aardvark does number of points as the phoenixx, then it doesril give magnifier to the eagle". so we can conclude 'the sardvark does not give a megnifier to the eegle". We know the serdvark does not give magnifier to the eagle and the sardverk does not esl the food of the spider, and according to Rule2 something does give magnifier the eagle and does not eat the food the spider, then knows defersive plans the tiger" so We know aardvark knows the defonsive plans of the tigor the procadlic does not ramove pleces of the tiger, and acconding Ru "rf the aardvark knows the defensive plans the tiger but the cracodite does not romove from the board one of the places the figer, then the tiger becomes an enemy of the cheotah", so we can conclude enamy cheatah" So statement "hhe fger becomas onamy of chestah" proved the tier "yes".
condlude "the nardwark
the defansiva plans the tiger". the board
the
Figure 26: Sample proofs from the dataset at depths 1, and 3. Higher depth proofs are substantially longer than the lower depth proofs.
cat, dog, plg, parrot, eagle, squireel, penguin, lion, tiger, donkey, leopard, cheetah, grizzly bear, polar bear, sun bear, panda bear, black bear, turtle, crocodile, elephant, panther. cow, rabbit. hare. buffalo, baboon, sheep, whale, jellyfish, carp, goldfish, viperfish, starfish, catrish, oscar, zander, sea bass, swordfish. salmon. hal- lout, bloblish, doctorfish, llapia, kangaroo, octopus, phoenix, aardvark, amberjack, eel, hummingbird, canary, hippopotamus, snail, caterpillar, mosquito, bat, ferret, gecko, kudu, moose, cockroach, cricket, grasshopper, meerkat. spider. lobster, squid, puftin, raven, kiwi, koala, wolverine
Train enti- ties
akita, bear, camel, coyote, snake, monkay, leopard, fish, ostrich, pigeon, dolphin. frog. goat, goose, wolf. go- rilla, beaver, lizard, flamingo, swan, elk, duck, reindeer, bison, shark, mouse, owl, llama, cobra, zebra, otter, crab, pealowl, rhino, dinosaur, dove, badger, chinchilla, cougar, crow, seal, worm, ant, bee, butterfly, dragonlly, dragon, gadwall, mule, liger. german shepherd, bulldog. husky, poodle, chihuahua, dachshund, basenji, dalma- tian. mermaid, seahorse, fangtooth, dugong, walrus, vampire, stork, swallow, songbird, woodpecker. starling, mannikin, pelikan, beetle, finch
Test enti- lies
owe money give magnifier to, learn the basics of resource management from, know the defensive plans of, show all her cards to, prepare armor for, sing victory song for, need support from, respect, raise peace flag for, become, an enemy roll the dice for, hold the same number points as, offer a job to, wink at. steal five points from, knock down the tortress of, burn the warehouse of, eat the food of, attack the green fields whose owner is, proceed to the spot that is right after the spot of, remove one the pieces of
Train pred- Icates
lear down the castle that belongs to, bring an lank for, reveal a secret to, enjoy the company of. neglect. want see, swear to, refuse to help, manage convince, call, stop the victory of, dance with, shout at, smile at, pay money to, unite with, hug. destroy the wall constructed by: create one castle for, disarm, acquire photograph of, borrow one the weapons of, fall owner is, leave the houses occupied by. hide the cards that she has from, swim in the pool next to the house of, negotiate deal with, trade one of its pieces with, build power plant near green fields of, take over emperor of, capture the king of, surrender to
Test predi- cales
square of, suspect the truthfuiness of, invest the company whose
Train tem- plates (sampled)
Universally quantified rule: something [A] the [B], then does not [C] the [D]
Existentially quantified rule: If at least one animal [A] the [B), then the [C] [D] the [E]
No Quantifier: For the [A]. the belief is that the [B] does not [C] the [A] and the [D] does not (E) the [A],
then you can add "the [A] does not [F] the [G] your conciusions.
No Quantifier: In order to conclude that the [A] does not [9] the (C]. two pieces of evidence are required: firstly
Test tem- plates (sampled)
that the [D] will not [E] the [A] and secondly the [F] [G] the [A]
Existential Quantifier: There exists an animal which [A] the [B]? Then, the [C] definitely does not [D] the [F].
Universal Quantifier: From observing that one animal [A] the [B), one can conclude that it also [C] the [D).
undoubtedly.
Table 3: Categories, descriptions, and examples of incomplete information in BoardgameQA. For lexical entailment, world knowledge, event times, and affordance, list of examples is written manually from which the sampling procedure can select. In the other cases, examples are generated automatically.
23


Reviews:
[{"review_id": "yeRGFL5g96", "reviewer": "Reviewer_cpZF", "summary": "", "questions": "", "limitations": "As stated before, I don’t think it is possible to highlight any limitations. Different ways of expanding further the dataset are also discussed in the paper as future work.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The current paper nicely covers all the required explanations and information for a work of this kind. Therefore, the main strengths I could highlight are:\n\n- The paper tackles an important task for current LMs, devising a dataset that outperforms previous ones.\n\n- The authors provide a detailed explanation of related work, focusing specifically on previous datasets with the same purpose, and their caveats.\n\n- The creation of the dataset is adequately explained and, at least for me, stresses further the validity and interest of the data presented in the current paper.\n\n- Beyond, the dataset is quite rich, in terms of the criteria considered for its creation, the different levels of complexity, the different types of incomplete information, etc.\n\n- The list of experiments is exhaustive, tackling many different questions of interest for the understanding of how LMs operate to solve the task of automated reasoning with incomplete information. \n\nFor all these reasons, I believe the paper is quite strong, and the dataset presented can provide a lot of value to the NLP community, and help improve LMs in such challenging and relevant task.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "sW4JJV6F4K", "reviewer": "Reviewer_Gau4", "summary": "", "questions": "", "limitations": "The paper does have a dedicated limitation section (but in the Appendix which I missed initially). It would be worth presenting these limitations in the main paper. \n", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "- A new benchmark dataset emulating reasoning in real-world scenario with conflicting information and reasoning with implicit knowledge \n- Broad empirical comparisons to show the challenging nature of this task\n- Careful analysis to highlight the key challenges of this benchmark (contradictions, type of conflicts, etc.) \n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "osYDCteOTb", "reviewer": "Reviewer_ayn7", "summary": "", "questions": "", "limitations": "The authors discussed the limitations in the appendix.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The BoardgameQA dataset is of high quality and might be meaningful for the research in reasoning with contradictory information.\n\n2. The experiments are comprehensive, and the results lead to several interesting findings.\n\n\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "AMaNwU6GhU", "reviewer": "Reviewer_MReC", "summary": "", "questions": "", "limitations": "Yes, the authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- It is important to analyze the various reasoning abilities of existing and widely used LMs. Here, the authors proposed a dataset to measure the reasoning ability with conflicting sources guided by preferences over sources (defeasible reasoning). \n\n- This paper proposed an algorithm to generate dataset in an effective way for different depths of reasoning. \n\n- In this work, the authors conducted a thorough analysis of different types of models in various experimental setups on BoardgameQA and presented several interesting findings about models’ defeasible reasoning ability. The findings suggest that existing LMs do not fare well for preference understanding and defeasible reasoning indicating the scope for developing more robust models for reasoning over contradicting and incomplete sets of information.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper presents BoardgameQA, a new benchmark dataset designed to evaluate LLM’s automated reasoning capabilities when there is conflicting, incomplete, or inconsistent information, i.e. QA problems contain facts and rules that can lead to conflicting conclusions. The experiments show that out-of-the-box LLMs do not handle these situations very well and finetuning does not seem to resolve the problem much, albeit small improvement. All the reviewers seem to agree that this dataset is a positive addition to the NLP community and all the reviewers hold a consensus that experiments and analysis are thorough.

There were comments that could further improve the paper. For example,
- max-depth 3 is too small for a synthetic dataset,
- conflicts in facts vs. conflicts via rules,
- comparison to the related work,
...

I believe the authors are on the way to generating max-depth>3 and already compared to more related work in Table 1. I recommend the authors to resolve/reflect comments like this in the camera ready.
Decision: Accept (Poster)