SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theory generation: The theory generation is the main component of the dataset generation that constructs the facts, rules and question to be used in each example. A high-level description is provided in Algorithm 1 and an example generation is shown in Appendix C. We first sample some sub-questions  $\mathcal{Q} = \{q_1,\dots ,q_n\}$  and a rule  $r$  which has  $\mathcal{Q}$  in its body and  $q$  in its head, such that  $q$  can be derived from  $\mathcal{Q}$  and  $r$ . The sampling is done by first selecting one of the aforementioned rule types, then matching the head of the rule to the question  $q$ , and then sampling sub-questions  $\mathcal{Q}$  based on the body of the rule. For example for the question! (dog, attack the fields, lion), we might sample the first rule type (see the six types above), then  $p_2$  will be mapped to attack the fields and  $e_2$  will be mapped to lion, and we also sample a sub-question such as (dog, unite with, cat) and add the rule  $\forall X:(X,\text{unite with},\text{cat})\Rightarrow!(X,\text{attacks the fields},\text{lion})$  to our set of rules. We then make a recursive call for each  $q_i$  to generate new rules and facts for them.
We then decide whether a conflict should be introduced or not, by using a biased coin flip with  $p_{Conf}$  representing the probability of conflict. If the decision is to produce conflicts, then we generate another set of sub-questions  $Q' = q_1', \ldots, q_m'$  and another rule  $r'$  such that!  $q$  can be derived from  $Q'$  and  $r'$ . Then we probabilistically decide if we want to generate a Type1 or a Type2 conflict using a biased coin flip with probability  $p_{ConfType1}$ . If the first case is selected, then  $r > r'$  is added to the preferences. In this case, we can make recursive calls for all or a subset of the facts in  $Q'$ . Otherwise,  $r' > r$  is added to the preferences. In this case, we make recursive calls for all but one of the facts in  $Q'$  (selecting randomly) to ensure that  $r'$  does not activate.
Proofs: We keep track of the facts, rules, and preferences during the generation process and turn them into proofs for the examples.
Stopping criterion: Every time we make a recursive call to the function in Algorithm 1, the example will contain one extra hop in its proof. We set the stopping criterion as the number of hops in the proof. Toward this goal, we included an argument  $d$  in Algorithm 1 which corresponds to the target maximum number of hops in the proof;  $d$  decreases by one every time we make a recursive call. When the algorithm is called with  $d = 0$ , instead of generating rules and sub-questions for the input question  $q$ , we simply add  $q$  to our set of facts.
Incomplete information: We generate examples with incomplete information where part of the knowledge should come from the LM (corresponds to rule type 5). For a question  $q$  in the theory generation phase, we sample sub-questions  $\mathcal{Q}$  and rule  $r$  such that  $\hat{\mathcal{Q}}$  can be derived based on  $\mathcal{Q}$  and  $q$  can be derived from  $\hat{\mathcal{Q}}$  and  $r$ . We then hide  $\hat{\mathcal{Q}}$  from the model so the model has to derive it itself. Algorithm 2 describes the procedure. We use a separate body of world knowledge, commonsense knowledge, mathematical, and orthography reasoning for generating  $\mathcal{Q}$  and  $\hat{\mathcal{Q}}$  (see Table 2 for a high-level description and Appendix C.2 for more details). For example, for the goal "the dog unites with the cat" we generate the sub-question "The dog is in Montreal." and the rule "If the dog is in Canada, then the dog unites with the cat." Then, an extra reasoning step is needed from the model to recognize that Montreal is in Canada.
We generate sub-questions and rules that require extra knowledge and reasoning with probability  $p_{MissInfo}$ ; otherwise, we create sub-questions and rules that require no extra knowledge and reasoning. To make the problem more challenging, we only include some categories of extra knowledge and reasoning in the training set; this ensures that the models cannot simply learn the extra knowledge from the training set and use it in the test set.
Conversion to natural language: Finally, once we generate the facts, rules, preferences, and question, we use manually constructed templates to turn each of them into a textual format. To make the problem more challenging, we use multiple templates per rule type and use some of the templates only in the test set (see Appendix C.3 for details).
A comparison of BoardgameQA with other prominent deductive reasoning datasets in terms of the average length of examples and the average number of unique tokens per example is provided in Figure 3.
Figure 3: A comparison of BoardgameQA with ProofWriter [48] and PrOntoQA [41] in terms of average length of examples and average number of unique tokens per example on depth 3 of the datasets.
Figure 4: The model performances on depths 1-3 of the BoardgameQA dataset. Many models struggle on this dataset, especially with higher depths.
Disproved and unknown examples: So far, we described how to generate examples with the label proved. Generating examples with the label disproved can be done simply by first generating an example with the label proved and then negating the question. Also, generating examples with the label unknown can be done by perturbing the theory until the statement in the question cannot be derived from the theory (e.g., reducing the amount of money of the frog to 50 dollars in the example of Figure 2). We randomly select and apply the following perturbations to the theory and run a defeasible solver implemented based on the scalable solver in [28] on the resulting theory until the label becomes unknown: 1- change the predicate of a fact or a rule, 2- change the sign of a fact or an element of the rule, 3- replace a fact with a new fact, and 4- flip the order of a preference.
One of the primary goals of our experiments is to verify if LMs are capable of reasoning in a defeasible setup. For this reason, we conduct experiments with various LM architectures (encoder-only, encoder-decoder, and decoder-only) and various pre-training and learning paradigms (finetune with and without proofs, prompt tuning, few-shot in-context learning, and instruction-tuned). Specifically, we test 1) finetuning BERT-large [14] with a classification head to predict the label directly, 2) finetuning T5 1.1 XXL [36] to generate the entire proof and then the label, 3) few-shotting PaLM 62B and PaLM 540B [9] where we provide demonstration examples and chain-of-thought (CoT) in the prompt (the CoT corresponds to the proof), 4) few-shotting the instruction-finetuned FLAN-PaLM 540B [10] with CoT, and 5) soft prompt-tuning [26] PaLM 62B with CoT where instead of providing a static prompt, we make the prompt embedding learnable and tune its parameters using the training data (the rest of the LM parameters are frozen). We report classification accuracy as the metric. We also report the majority class baseline ( $\sim 33\%$  since our labels are balanced).
Dataset sizes: To gain a more detailed understanding of the models' defeasible reasoning capacity, we create several variations of BoardgameQA. The nature of the variation will be discussed in the remainder of this section with each experiment. For each variation, we sample 1000 examples for train, 500 for validation, and 1000 for test. We sample an equal number of examples from each label.
5.1 Can LMs Reason with Contradictory Inputs?
As explained in Section 4, BoardgameQA makes use of a number of variables that control various aspects of the dataset such as the amount and types of conflict and the amount of extra knowledge required. We start by creating a default version of the dataset that exhibits each of these properties to some degree by setting  $p_{\text{Conf}} = 0.5$ ,  $p_{\text{ConfType1}} = 0.5$ , and  $p_{\text{MissInfo}} = 0.5$ . We then generate three datasets with depth 1-3 (i.e., requiring 1-3 hop(s) of reasoning, respectively), and measure the performance of our baselines on these datasets.
The results are in Figure 4. The tuned models perform reasonably on depth 1, but their performance substantially degrades on depths 2-3. This contrasts with previous observations for monotonic reasoning (e.g., in [11, 48]) where finetuned LMs reach near-perfect performance even on higher depths. This indicates that reasoning with contradictory inputs is more difficult even with finetuning. Moreover, we see that the few-shot models perform poorly across all depths showing that conflict resolution is not achieved out-of-the-box with pretrained models. This includes both PaLM and instruction-finetuned FLAN PaLM models. PaLM 540B performs better than PaLM 62B showing that
larger models may have higher capacity for defeasible reasoning. More insights from full confusion matrices can be found in Appendix A.
Hereafter, due to inference costs, we only experiment with finetuned BERT and T5, prompt-tuned PaLM 62B, and few-shot PaLM 540B, and with examples of depth 2 to keep a medium level of difficulty in terms of reasoning hops and enable measuring the effect of the other factors.
5.2 Does Correct Label Prediction Mean Correct Proof?
Recently, it has been shown that although large LMs achieve high accuracy on label prediction for (monotonic) reasoning task, they do so by generating spurious proofs that do not represent valid steps of reasoning [23]. There is also evidence that LMs frequently exploit spurious correlations in the data distribution to achieve high label accuracy, rather than reasoning purely deductively [59]. Hence we design evaluation metrics to reflect a more rigorous measure of accurate defeasible reasoning. In the case where a model predicts the label correctly, and the label is one of proved or disproved (where an actual proof exists), we measure whether the proof generated by the model is correct or not. For this purpose, we compute two automated proof accuracy metrics (named Rule F1 and Conflict F1) and one manual metric (named Overall Proof Accuracy) as described below. For Rule F1, we extract the rules used in the golden proof and the ones in the proof generated by the model that are used to derive new facts (and ultimately, the goal). Then we compute the F1-score of the overlap of the two sets. For Conflict F1, we extract the conflict resolutions (corresponding to pairs of rules) used in the gold proof and the ones in the proof generated by the model, and compute the F1-score of their overlap. For Overall Proof Accuracy, we manually verify whether the proof is correct for 50 sampled examples per model. We compute these metrics on depth 2 of the dataset.
According to the results in Figure 5, all models perform relatively well in selecting the correct set of rules for the proof. The few-shot model performs poorly on conflict resolution whereas the tuned models perform substantially better, suggesting that preference understanding and conflict resolution do not surface with simple few-shot prompting, and tuning is required for models to exhibit this capacity. Second, the models often generate wrong proofs, even when they predict the label correctly. The issue is less severe in the case of the prompt-tuned model but becomes more severe
Figure 5: Proof accuracy metrics for various models on depth 2 of the dataset, when the label is predicted correctly.
for the finetuned and few-shot models. We provide examples of proof failures in Appendix A.
5.3 Do Conflicts Make Reasoning More Difficult?
We create four versions of BoardgameQA named NoConflict, LowConflict, MediumConflict, and HighConflict, with  $p_{Conf}$  set to 0.0, 0.2, 0.5 and 0.8 respectively; other factors are kept the same. Note that the MediumConflict corresponds to the dataset in Figure 4. The results of the models on these datasets are reported in Figure 6. The performance of all models monotonically degrades as the number of conflicts increases, showing that conflict resolution is indeed a major factor in the difficulty of the problems. For
example, BERT performs above-random for the NoConflict and LowConflict cases, but the model performance drops to near-random on MediumConflict and HighConflict cases.
Figure 6: The model performances on four versions of the BoardgameQA dataset with various amounts of conflicts in them.
5.4 Which Conflict Type is More Difficult to Resolve?
To test which type of conflict (See sec. 4) is more difficult for the models, we create three versions of the dataset with varying proportions of Type1 vs Type2 conflicts, by setting  $p_{ConfType1}$  to 0.2, 0.5, and 0.8 respectively. The first dataset mostly contains conflicts of Type1, the second contains both
conflicts in a similar amount, and the third dataset contains mostly Type2 conflicts. The other factors are kept constant across the datasets.
The results of the models are reported in Figure 7. We see that models perform slightly better on the dataset with mostly Type1 conflicts. This discrepancy between performance on Type1 and Type2 conflicts is intuitive because in the case of Type1 conflicts, the model can ignore the conflicting rule and whether its body can be proved, but in the case of Type2 conflicts, the model has to show that at least one of the elements in the body of the conflicting rule cannot be proved. In the case of tuned models, we furthermore observe that biasing the dataset toward one conflict type results in better performance
overall. This might be because the model mostly needs to learn to resolve one type of conflict which may be easier than learning both.
Figure 7: The model performances on three versions of the BoardgameQA dataset with different distributions on the type of conflicts.
5.5 Does Information Incompleteness Make Reasoning More Difficult?
As described in Section 4, we can control the amount of information incompleteness using a parameter which we named  $p_{MissInfo}$ . To test how the information incompleteness affects the performance of various models, we create three versions of our dataset with  $p_{MissInfo}$  set to 0.2, 0.5 and 0.8, which we name KnowledgeLight, KnowledgeMedium and KnowledgeHeavy, respectively.
The results are reported in Figure 8. We observe that as the amount of required knowledge increases, the performance of the fine-tuned models decreases accordingly. However, the performance of the prompt-tuned and few-shot models remain relatively unchanged, likely due to the larger size of the model and the extra amount of knowledge that is present in the model, as well as the fact that working with real-world knowledge might be easier for these models than with artificial knowledge.
Figure 8: The model performances on three versions of BoardgameQA with various degrees of incomplete information.
5.6 Do Distractors Make Reasoning More Difficult?
We also measure the effect of distracting facts and rules on model performance. A distracting fact or rule is one that does not appear in the proof and does not change the label. In Figure 2, for example, "the frog has a knife" is a distracting fact. To this end, each time we call Algorithm 1, besides the sampled subquestions, we also sample some distracting sub-questions and add them to the set of subquestions. We create three versions of the BoardgameQA dataset where we add 0, 1, and 2 distracting facts in each step, which we
name NoDistractors, SomeDistractors, and ManyDistractors, respectively.
Figure 9: The model performances on three versions of BoardgameQA with various amounts of distracting facts and rules.
According to the results in Figure 9, the performance of the tuned models does not substantially degrade with a small number of distractors, potentially because the distractors can help the model avoid learning spurious correlations. However, their performance drops substantially with more distractors. Also, with more distractors, the performance of the few-shot model decreases monotonically,
although only marginally (this observation is consistent with the results of [42]). This shows that distractors (that are typically common in real applications) can also compound the problem difficulty.
Our dataset, in its current form, focuses primarily on deductive logical entailment, where the problem is a classification problem (label  $\in$  {proved, disproved, unknown}), and the contradictions are also binary (i.e. one rule suggesting something is True and the other suggesting it is False). Future work can extend BoardgameQA and the analysis provided in this work to non-classification cases where 1- one needs to apply defeasible logical reasoning to answer questions such as "Who will be attacked by the dog?", 2- one needs to resolve non-binary conflicts where, e.g., one rule suggests "the dog is currently in Canada" and the other suggests "the dog is currently in Australia", 3- there are conflicts and preferences over facts as well, e.g., Fact1: Fiona has travelled to every country in Europe, Fact2: Fiona has not travelled to the Scandinavian countries, Fact2 is preferred over Fact1.
The current work assumes the initial state (facts) and the rules of the game are small enough to be included in the prompt. It is also limited to deductive reasoning with the modus ponens rule. Future work can extend BoardgameQA and our analyses to the cases where not all the facts and rules can be included in the prompt due to the limitation in the prompt length, as well as the case with other types of rules such as proof by contradiction, disjunction elimination, etc (see [42]).
In this work, we only studied one simple but highly practical solution to conflict resolution (i.e. based on preferences). Future work can extend BoardgameQA and the analysis in this paper to other natural types of conflict resolution. Note that in some applications, preferences for conflict resolution have to be assigned with great care and diligence to avoid unfair treatment of information sources.
In this work, we introduced BoardgameQA, a dataset for measuring the natural language reasoning ability of language models (LMs) in the presence of conflicting input sources. Our dataset furthermore includes scenarios in which the knowledge required for reasoning is only partially provided as input and additional information needs to come from the model itself. We tested several types of LMs on different variations of the dataset and observed that LMs perform poorly when reasoning with conflicting inputs. In the case of smaller models, the performance was also poor when additional knowledge from the LM is needed. Since reasoning over contradicting and incomplete sets of information is a common scenario in real-world applications, our results highlight an important gap in the reasoning capacity of current LMs. We hope our dataset can guide future work developing methodology to improve the reasoning ability of LMs under this setup, or finding alternative formulations of conflict resolution that better facilitate LM reasoning.
We thank Yue Liu, Tania Bedrax-Weiss, and the anonymous reviewers for their valuable feedback.
[1] Allaway, E., Hwang, J. D., Bhagavatula, C., McKeown, K., Downey, D., and Choi, Y. Penguins don't fly: Reasoning about generics through instantiations and exceptions. arXiv preprint arXiv:2205.11658, 2022.
[2] Arabshahi, F., Lee, J., Gawarecki, M., Mazaitis, K., Azaria, A., and Mitchell, T. Conversational neuro-symbolic commonsense reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 4902-4911, 2021.
[3] BehnamGhader, P., Miret, S., and Reddy, S. Can retriever-augmented language models reason? the blame game between the retriever and the language model. arXiv preprint arXiv:2212.09146, 2022.
[51] Wang, B., Deng, X., and Sun, H. Iteratively prompt pre-trained language models for chain of thought. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 2714-2730, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022.emnlp-main.174.
[52] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., and Zhou, D. Chain-of-thought prompting elicits reasoning in large language models. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 24824-24837. Curran Associates, Inc., 2022.
[53] Weston, J., Bordes, A., Chopra, S., Rush, A. M., Van Merrienboer, B., Joulin, A., and Mikolov, T. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698, 2015.
[54] Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601, 2023.
[55] Ye, A., Cui, C., Shi, T., and Riedl, M. O. Neural story planning. arXiv preprint arXiv:2212.08718, 2022.
[56] Yu, F., Zhang, H., and Wang, B. Nature language reasoning, a survey. arXiv preprint arXiv:2303.14725, 2023.
[57] Yuan, Q., Kazemi, M., Xu, X., Noble, I., Imbrasaite, V., and Ramachandran, D. Tasklama: Probing the complex task understanding of language models. arXiv preprint arXiv:2308.15299, 2023.
[58] Zelikman, E., Wu, Y., Mu, J., and Goodman, N. STaR: Bootstrapping reasoning with reasoning. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 15476-15488. Curran Associates, Inc., 2022.
[59] Zhang, H., Li, L. H., Meng, T., Chang, K.-W., and Broeck, G. V. d. On the paradox of learning to reason from data. arXiv:2205.11502, 2022. URL https://arxiv.org/abs/2205.11502.
[60] Zhang, H., Li, Z., Huang, J., Naik, M., and Xing, E. Improved logical reasoning of language models via differentiable symbolic programming. In First Workshop on Pretraining: Perspectives, Pitfalls, and Paths Forward at ICML 2022, 2022. URL https://openreview.net/forum?id=81Ny3QCaxHX.
[61] Zhong, W., Wang, S., Tang, D., Xu, Z., Guo, D., Wang, J., Yin, J., Zhou, M., and Duan, N. AR-LSAT: Investigating analytical reasoning of text. arXiv preprint arXiv:2104.06598, 2021.
Figure 10: The model performances on various depths of a binary version of the BoardgameQA dataset.
Figure 11: Highlighting some of the model errors for a depth 1 example when using PaLM 540B.
Model Output: Since the vampire has more money than the shark and the bee combined and according to Rule1 if the vampire has more money than the shark and the bee combined then we can conclude that it disarms the bulldog and for the conflicting and lower priority rule Rule2 we cannot prove the antecedent there exists an animal which leaves the houses that are occupied by the liger so the vampire disarms the bulldog. So the statement the vampire disarms the bulldog is proved and the answer is "yes".
Error Analysis:
The model assumes the vampire has more money than the shark and the bee combined, but the amount of money the vampire has is not specified.
Since Rule1 has higher priority than Rule2, if the model believes Rule1 activates, there should be no need to show that Rule2 does not activate.
For Rule2, the model assumes that we cannot prove that there exists an animal which leaves the houses that are occupied by the tiger, whereas we know the dragon does this based on the facts.
A More Experimental Results and Analysis
Binary Classification: Kazemi et al. [23] observed that reasoning with unknown labels is particularly challenging for few-shot LMs, because providing a natural chain-of-thought for unknown is difficult. To measure if the poor performance is merely due to the existence of examples with unknown label or due to conflict resolution being difficult for these models, we also created a binary version of the dataset for depths 1, 2, and 3 where only examples with proved and disproved labels are included. The results are reported in Figure 10. We overall see similar patterns as the binary case, except for some improvements for the T5 model on depth 2.