LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models (2024)

Abstract: Recently developed large language models (LLMs) have been shown to perform remarkably well on a wide range of language understanding tasks. But, can they really “reason” over the natural language? This question has been receiving significant research attention and many reasoning skills such as commonsense, numerical, and qualitative have been studied. However, the crucial skill pertaining to ‘logical reasoning’ has remained underexplored. Existing work investigating this reasoning ability of LLMs has focused only on a couple of inference rules (such as modus ponens and modus tollens) of propositional and first-order logic. Addressing the above limitation, we comprehensively evaluate the logical reasoning ability of LLMs on 25 different reasoning patterns spanning over propositional, first-order, and non-monotonic logics. To enable systematic evaluation, we introduce LogicBench, a natural language question-answering dataset focusing on the use of a single inference rule. We conduct detailed analysis with a range of LLMs such as GPT-4, ChatGPT, Gemini, Llama-2, and Mistral using chain-of-thought prompting. Experimental results show that existing LLMs do not fare well on LogicBench; especially, they struggle with instances involving complex reasoning and negations. Furthermore, they sometimes overlook contextual information necessary for reasoning to arrive at the correct conclusion. We believe that our work and findings facilitate future research for evaluating and enhancing the logical reasoning ability of LLMs1.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13679–13707 August 11-16, 2024 ©2024 Association for Computational Linguistics
1 Introduction
Large language models such as GPT-4, ChatGPT, Google Gemini, Llama-2 (Touvron et al., 2023), and Mistral (Jiang et al., 2023) have made remarkable progress in NLP research enabling machines
1Data and code are available at https://github.com/ Mihir3009/LogicBench
to perform a variety of language tasks that were previously thought to be exclusive to humans (OpenAI, 2023; Brown et al., 2020; Zhao et al., 2023). However, the ability of these LLMs to reason “logically” over natural language text remains under-explored, even though logical reasoning is a fundamental aspect of intelligence and a crucial requirement for many practical applications, such as questionanswering systems (Khashabi, 2019) and conversational agents (Beygi et al., 2022). Although several datasets have been proposed (Clark et al., 2021; Tian et al., 2021; Joshi et al., 2020; Saeed et al., 2021) to evaluate the logical reasoning capabilities of LLMs, these datasets are limited in their scope by (1) not evaluating logical reasoning independently of other forms of reasoning such as LogiQA (Liu et al., 2021a) and ReClor (Yu et al., 2020); and (2) evaluating only a single type of logic and covering only few logical inference rules as done in FOLIO (Han et al., 2022) and ProntoQA (Saparov and He, 2023). Thus, our aim in this work is to address the lacuna of having a more comprehensive set of inference rules for evaluating the logical reasoning ability of LLMs.
To this end, we introduce a systematically created question-answering dataset for the evaluation of logical reasoning ability using a single inference
13679
rule, called LogicBench. Besides evaluating the logical reasoning ability of LLMs, by evaluating models on single inference rules, we can also gain insights into the frequency of text sequences corresponding to these rules in the pre-training data and their impact on model performance. As illustrated in Figure 1, LogicBench includes a total of 25 reasoning patterns across ‘propositional, first-order, and non-monotonic’ logics. To the best of the authors’ knowledge, this is the first work to study non-monotonic reasoning, as well as various inference rules in propositional and first-order logics including hypothetical and disjunctive syllogism; and bidirectional, constructive, and destructive dilemmas in NLP domain. To evaluate LLMs using LogicBench, we formulate two different tasks: (i) a Binary Question-Answering (BQA) task in which the context comprises logical statements and the models have to determine whether a conclusion given in the question is logically entailed by the context, and (ii) a Multiple-Choice Questions-Answering (MCQA) task where models must select the most appropriate logical conclusion from four distinct options, based on the provided context. The rationale behind having BQA and MCQA tasks is that it provides systematic standard metric-based evaluation (i.e., direct comparison of LLMs’ performance in terms of accuracy), which could be more challenging with open-ended question-answer formats. Examples instances of various reasoning patterns are presented in Table 4 and App. C.
To construct LogicBench, we use a three-stage procedure (refer to §3). In the first stage, we prompt GPT-3.5 to generate a variety of coherent natural language sentences having different ‘ontologies’ (i.e., a collection of concepts as car, person, and animals) and their corresponding negations (refer to §3.2.1). In the second stage, we generate (context, question) pairs where the context represents a natural language narrative consisting of logical statements, and the question is formulated to exhibit the logical conclusion derived from the context. In the third stage, we generate task-specific data instances (i.e., (context, question, answer) triplets).
We conduct a comprehensive evaluation with a range of LLMs on LogicBench including GPT-4, ChatGPT (GPT-3.5-Turbo), Gemini-Pro, Llama2-7B-Chat, and Mistral-7B-Instruct using chainof-thought (Wei et al., 2022). In particular, we measure the accuracy of LLMs predictions on both BQA and MCQA tasks. Our experiments result in several interesting findings such as LLMs often
struggle to reason over complex logical contexts that involve only a single inference rule and encounter difficulties with inference rules involving negations. Experimental results reveal that these models struggle with respect to many of the inference rules and patterns, suggesting significant room for improvement in their logical reasoning abilities. To further demonstrate the use of LogicBench, we synthetically augment it and fine-tune T5-large. Our preliminary results (App. I) show that this improves the logical reasoning ability of existing models leading to performance improvement on other logic datasets, LogicNLI, and FOLIO (∼ 2% on an average), and shows competitive performance on LogiQA and ReClor.
2 Related Work
As LLMs continue to evolve rapidly, it becomes increasingly crucial to evaluate their diverse reasoning capabilities, as well as those of forthcoming LLMs. LogiQA (Liu et al., 2021a) and ReClor (Yu et al., 2020) have made notable contributions by compiling multichoice questions from standardized examinations that demand diverse forms of logical reasoning. In contrast to LogicBench, these datasets involve mixed forms of reasoning and do not focus on assessing logical reasoning in isolation.
A few past attempts have been made to evaluate only logical reasoning while excluding other forms of reasoning. For example, CLUTTER (Sinha et al., 2019) covers inductive reasoning, (Hahn et al., 2021) covers temporal logic, and Ruletaker (Clark et al., 2021) evaluates whether a transformerbased model emulates deductive reasoning over synthetically generated statements in a limited setting. LogicNLI (Tian et al., 2021) introduced a diagnostic benchmark for FOL reasoning, with the dataset constructed by automatically generating logic expressions and replacing the entity and attribute placeholders.
Our proposed dataset is similar (in terms of task formulation) to ProofWriter (Tafjord et al., 2021),
FOLIO (Han et al., 2022), and ProntoQA (Saparov and He, 2023) which are QA datasets designed to test reasoning ability. ProofWriter provides multihop proofs for each example, while FOLIO gives diverse and complex logical expressions, however, it is only limited to FOL. ProntoQA (Saparov and He, 2023) provides explanation and reasoning steps but is limited to modus ponens in FOL. Nevertheless, several crucial attributes motivated us to create LogicBench (see Table 1 for comparison). Additional datasets for evaluating logical reasoning also exist such as SimpleLogic (Zhang et al., 2023) provides a class of logical reasoning problems, TaxiNLI (Joshi et al., 2020) introduces logical taxonomy in NLI task and RuleBert (Saeed et al., 2021) covers only soft logical rules. In summary, LogicBench evaluates logical reasoning in isolation and provides diverse inference rules and logic types compared to existing datasets. Extended related work is discussed in App. B.
3 LogicBench
3.1 Logics Types
Propositional Logic (PL) Propositional logic employs a collection of statements or propositions (denoted as P = p1, p2, ..., pn, where pi represents a proposition) and builds upon them using logical connectives such as ‘∧’, ‘∨’, ‘→’, ‘↔’, and ‘¬’. Several inference rules for propositional logic have been defined using which given a set of premises, one can derive a sound conclusion. For example, let’s consider two propositions: p1, which states "It is raining," and p2, which states "It is cloudy." From these propositions, we can
construct a context/knowledge base (KB) with two premises: (1) p1 → p2 and (2) p1. With this KB, we can conclude p2. This inference rule is written as ((p1 → p2) ∧ p1) ⊢ p2 and is known as ‘Modus Ponens’. In our study, we explore nine distinct inference rules of propositional logic, extensions of seven of them with one-variable and a universal quantifier, and two axioms of first-order logic as shown in Table 2. These inference rules provide a proper framework for deriving valid conclusions.
First-order Logic (FOL) In this work, we consider a restricted set of logical axioms for FOL that utilize quantifiers, ∀ (universal quantifier) and ∃ (existential quantifier). The universal quantifier (∀) denotes that a statement holds true for all instances within a specific category. In contrast, the existential quantifier (∃) indicates that a statement is true for at least one instance within its scope. For instance, a simple extension of propositional ‘Modus Ponens’ is an inference rule where given the premises ∀(p(x) → q(x)) and p(a), we conclude q(a) (e.g., given “All kings are greedy” and “Sam is a king”, we can conclude “Sam is greedy”). Here, we explore two axioms (EG and UI - in detail in App. C.3) and various inference rules that incorporate the quantifiers (shown in Table 2).
Non-monotonic (NM) Reasoning In this work, we analyze a range of logical reasoning templates in NM logics involving “Default Reasoning,” “Reasoning about Unknown Expectations,” and “Reasoning about Priorities.” These templates are inspired by the compilation (Lifschitz, 1989) made in 1989 to evaluate the abilities of various non-
monotonic logics that were being developed at that time. Below Table 3 shows examples of NM reasoning. Additional examples are given in App. C.4.
A key aspect of NM logics is to formalize notions such as “normally,” “typically,” and “usually” that are not directly formalizable using classical quantifiers in the first-order setting. The general rule “Heavy blocks are normally located on the table” does not imply that “All heavy blocks are always located on the table”. Rather, this rule allows for exceptions. Our work explores various NM reasoning patterns, as depicted in Figure 1, to delve deeper into the nuances of this type of reasoning.
3.2 Data Creation
Our data creation procedure, illustrated in Figure 2, consists of three stages:
1. Sentence Generation: Starting with a given prompt, we generate coherent sentences and their negations that incorporate different ontologies.
2. NL Conversion: Pairs of (context, question) are generated using pre-defined templates from which context is then converted to a natural language narrative using the prompt.
3. Task Instance Generation: Task-specific (context, question, answer) triplets are generated. BQA requires answers in the form of “yes” or “no”, whereas MCQA involves selecting one correct option from a set of four. We generate semantically preserving and inverting variations of these triplets to add more diversity for BQA.
Examples of generated data corresponding to each logic type and reasoning patterns are presented in App. C.
3.2.1 Sentence Generation Here, the first step is to generate sentences with diverse ontologies. An ontology represents a col-
lection of concepts (e.g. car, person, animals, etc.) along with their corresponding associated properties. To generate these sentences, we prompt the GPT-3.5 model with instructions tailored for each inference rule (more details in App. A).
An example of a prompt corresponding to the ‘Modus Tollens’ from PL is presented in App. A for better illustration. Note that our objective at this stage is not to generate logical sentences but rather to generate a diverse and coherent set of sentences that encompass various concepts. We also create a negation sentence corresponding to each generated sentence2. In this work, the scope of generating negations is simple (refer to App. C for examples), however, negations can be more complicated in the case of logic. These generated sentences will be combined with logical connectives in a later stage to form context and questions.
3.2.2 NL Conversion Here, the NL conversion is accomplished using two steps. First, we leverage the formal expressions of reasoning patterns to create templates that establish the desired NL formulation for each logical connective (i.e., templatized context). Second, we prompt GPT-3.5 to transform the templatized context into a story/narrative-based context, enhancing its naturalness. For instance, implication: “p → q” is expressed as “If p, then q”, conjunction: “p ∧ q” as “p and q.”, and disjunction: “p ∨ q” as “At least one of the following is true: (1) p and (2) q. Note that we do not know which of (1) and (2) is true. It is possible that only (1) is true, or only (2) is true, or both are true.” since understanding the logical implication of ‘or’ when integrated into logical formulations posed challenges to both humans and
2We use https://github.com/dmlls/negate to generate negated sentences
models. With these established formulations, we proceed to utilize the sentences generated in §3.2.1 to create the templatized context and questions corresponding to reasoning patterns.
Then, the templatized context is converted into a narrative-based context using a prompt-based converter, enhancing its naturalness. The promptbased converter (essentially, prompting GPT-3.5) ensures that the context is no longer templatized yet follows the logical connection between sentences as mentioned in the logical rule (further details are presented in App. D). For instance, let’s consider the “Modus Tollens” from PL (((p → q) ∧ ¬q) ⊢ ¬p), and the “Bidirectional Dilemma” from FOL (∀x((p(x) → q(x))∧(r(x) → s(x)))∧ (p(a) ∨ ¬s(a))) ⊢ (q(a) ∨ ¬r(a))). For these rules, Table 4 presents examples of logical templatized and narrative-based context, question, and task instances for both, BQA and MCQA. App. D showcases further examples corresponding to each inference rule and patterns from LogicBench.
3.2.3 Task Instance Generation
After generating the context and questions in §3.2.2, we generate (context, question, answer) triplets for both tasks: (i) BQA, and (ii) MCQA. Here, narrative-based context is similar for both BQA and MCQA tasks, only the format of (question, answer) pairs are different.
BQA We generate semantically preserving and inverting variations of questions. Let’s consider the example of “Modus Tollens” from Table 4, having question as: “Does this imply that Liam didn’t
finish his work early?” In this question, we observe one proposition: s1, representing the statement “Liam didn’t finish his work early,” can be used to create another question that did not follow the logical rule MT. We can create two possible tuples: < ¬s1, yes >,< s1, no >. Each tuple has a question-answer combination using proposition s1. Moreover, we do not generate variations for the context since it offers no substantial diversity in the dataset. For question variations, we replace the variation of proposition < s1 > in the original question with the corresponding tuples to add diversity to LogicBench. The process allows us to create more variations of the question for BQA tasks, as illustrated in Figure 2 (Step 3 - Task 1).
MCQA A prompt-based approach is used to create different incorrect options. For rule MT, as shown in Table 4, option 1 is a correct option that follows the rule MT logically (< ¬s1 >), while the other three options are generated using prompting in a way that does not follow the rule being incorrect options. In addition to the options, the question is also replaced by a randomly selected question from the set of five questions. As seen in Figure 2 (Step 3 - Task 2), there is only one correct option out of four given options. More details related to incorrect option generation and instances examples for both tasks are in the App. C, and E.2.
3.3 Statistics and Qualitative Analysis
Statistics We introduce two versions of our proposed dataset: LogicBench(Eval)BQA and LogicBench(Eval)MCQA. Statistics of both ver-
sions are presented in Table 5. For LogicBench(Eval)BQA, out of 1520, 520 samples are for ‘yes’ and 1000 samples are for ‘no’ labels. For LogicBench(Eval)MCQA, there are 20 unique samples present for each rule, thus in total 500 unique samples. Furthermore, we synthetically augmented LogicBench(Eval) for training purposes (i.e., LogicBench(Aug)) which consists of 150 unique data samples for each rule for BQA, resulting in a total of 12908 data samples including variations.
Data Validation Throughout the data generation phase of LogicBench(Eval), the authors conduct a review of the logical formations to ensure they follow the intended logical structure. We examine each narrative for any potential discrepancies, ensuring that they are logically sound and correctly represent the intended relationships between propositions. In addition to the logical formation, we also dedicated considerable effort to eliminating typos and validating the grammar. We also analyze the diversity in terms of different ontology and the logical nature of the LogicBench(Eval) (presented in App. C.1). We mitigate errors encountered during the validation step (presented in App. F).
4 Results and Analysis
4.1 Experimental Setup Task Formulation For BQA, let us consider a set of data instances Ir,L corresponding to the inference rule r and logic type L. In this set, ith instance
is represented as Iir,L = {(ci, Qi)} where ci represents narrative context and Qi = {q1, q2, ..., qn} represents set of question and its variations corresponding to ith instance. As discussed in §3, each context (c) represents logical rules (e.g., All cats have fur. Tom is a cat.) and question (q) represents the conclusion (e.g., Does Tom have fur?). To each context and question pair, i.e., < c, q >, we assign a label from the set Y = {Y es,No}. We assign a label Y es if the conclusion logically entails the context, otherwise, assign a label No. To evaluate any LLMs on this setup, we provide < p, c, q > as input to predict a label from Y where p is a natural language prompt. In the set Ir,L for MCQA, ith instance is represented as Iir,L = {(ci, qi, Oi)} where ci represents narrative context and qi represents question and Oi = {o1, o2, o3, o4} represents four option choices. To each context and question pair, i.e., < c, q >, we assign a label from the set Y = {o1, o2, o3, o4}. We assign a label o1 if the correct conclusion is presented in the first option, and likewise for other labels. To evaluate any LLMs on this setup, we provide < p, c, q, o > as input to predict a label from Y .
Experiments We evaluate a range of prompting models including GPT-4, ChatGPT (GPT-3.5Turbo), Google Gemini-Pro, Llama-2-7B-Chat, and Mistral-7B-Instruct-v0.2. Each model is evaluated in a zero-shot setting where the chain-ofthought prompt is provided to the model without any in-context examples. This approach allows us to determine LLM’s inherent ability to do logical reasoning (based on pre-training), as we can not expect that various logical inference rules/patterns will always be made part of prompts. However, we
do evaluate these models in a few-shot setting, and present the results in App. G.
Metrics Here, we evaluate performance in terms of accuracy for both tasks, BQA and MCQA. For the BQA, we measure accuracy corresponding to each label, i.e., A(Y es) and A(No). We evaluate each model on three different chain-of-thought prompts and report average results across these prompts. All prompts used for experiments are described in App. H.
4.2 Main Results
Table 6 and Table 7 represent inference rule-wise performance corresponding to each LLMs for the BCQ and MCQA tasks, respectively. Specifically, Table 6 provides label-wise accuracy (A(Y es) and A(No)) for the BQA task, and Table 7 provides overall accuracy for the MCQA task. Both tables provide valuable insights into the performance of different models on various logic types and lead to several interesting findings. From Table 6, we
can observe that ChatGPT achieves 48.04%, and GPT-4 shows a performance of 63.98% A(Y es) on average which indicates the challenge of classical logical reasoning (PL) even for larger LLMs such as ChatGPT and GPT-4. Furthermore, we can observe that models struggle more with inference rules of PL compared to FOL and NM. In addition, it is noticeable that each model performs relatively better on questions with a negative response (i.e., No) compared to questions with a positive response (i.e., Y es). This observation suggests that the models struggle to fully comprehend the logical relationship between the context and the conclusion (i.e., lower A(Y es)). However, they demonstrate a relatively stronger understanding when the relationship is contradictory in nature (i.e., higher A(No)). From Table 7, we can observe that larger models exhibit superior performance in selecting the correct choice to arrive at a logical conclusion. Interestingly, the performance of models decreases when the inference rules are longer or include nega-
tions. In contrast to Table 6, for MCQA, LLMs show superior performance for PL and FOL compared to NM. To further investigate these findings and provide a detailed analysis, we perform a thorough study of reasoning chains generated by LLMs and present our insights in the subsequent section.
4.3 Analysis and Discussion
Human Performance We conduct a human evaluation on a subset of LogicBench(Eval) for both tasks, BQA and MCQA. Specifically, we selected 50 unique instances covering all 25 reasoning patterns from LogicBench(Eval). This selection resulted in total instances of 153 <context, question> pairs for BQA, and 50 <context, question, choices> pairs for MCQA. We hired three graduate student volunteers to provide the evaluations. The task instructions given to all three annotators closely resemble the prompts provided to models (App. H). Each instance pair is answered/annotated by three different annotators with 0.785 interannotator agreement (measured with raw/observed agreement) for BQA and 0.813 for MCQA.
From the results (Table 8) for BQA, we see that humans achieve more than ∼ 85% accuracy on various logic types on LogicBench(Eval) which indicates the capability of humans to comprehend single-step logical reasoning effectively. From Ta-
ble 6, we observe that the average performance of all models is below human performance indicating room for improvement in their reasoning capabilities. From Table 7, we make similar observations for MCQA. However, we can see that the performance of NM for the MCQA task remains a challenge for both humans and LLMs.
Lower performance of LLMs on PL as compared to NM and FOL for BQA. In the development of AI, NM logic was partly developed to formalize natural language constructs, such as “normally birds fly”, that were not formalizable in a straightforward manner using classical mathematical logics. Thus, while it was difficult for researchers to come up with non-monotonic logics and formalize non-monotonic reasoning, the fact that they were usually motivated by natural language examples, suggests that many of the nonmonotonic reasoning aspects are present in the NL text in the wild that is used in the pre-training of the ultra-large LLMs such as GPT4. While from human experience and complexity theory, FOL is harder than PL in general; in the LLM context, the crucial factor becomes what kind of logical sentences LLMs are pre-trained on. It seems that LLMs are pre-trained more on simple FOL sentences than on simple PL sentences (see Appendix I for further discussion). On the other hand, some PL features are perhaps less prevalent in human writing (on which LLMs are pre-trained) - such as Modes Tollens. Table 6 shows that GPT-4 achieves ∼ 85% accuracy (A(Y es)) for simple inference rules such as HS(PL). However, GPT-4 performance dropped to ∼ 59% A(Y es) for PL(MT).
Negations are hard to understand when embedded with logical rules. Regarding PL and FOL, it is apparent that the models struggle more with the DD and MT inference rules. A closer look at Table 2 reveals that all of these rules include examples where the models need to draw conclusions based
on negated premises. This indicates that the models encounter difficulties when negated premises are introduced. We also analyze the effect of negations on the reasoning chain (see App. I).
Longer inference rules are still challenging. Table 6 indicates that the models face challenges when handling longer rules, such as BD, CD, and DD, both in PL and FOL. Hence, it can be concluded that these models struggle with longer logical dependencies in the premise, particularly when a higher number of propositions are present. In the case of NM reasoning, the models exhibit lower performance in DRS, indicating that a higher number of premises often leads to more frequent mistakes.
LLMs sometimes overlook contextual information. We investigate the LLMs’ logical reasoning ability in natural language, not in artificial logical formulations. Hence, we note that LLMs sometimes hallucinate information and overlook contextual information, leading to incorrect conclusions. To analyze this, we manually examine all the reasoning chains generated for instances sharing the same contexts in both BQA and MCQA tasks. We observe that, although this pattern is not dominant, it affects BQA more than MCQA. For a more indepth analysis, please refer to App. I.
Large models are better logical reasoners. We analyze the results of both smaller (Llama-2-7B and Mistral-7B) and larger (ChatGPT, GPT-4, and Gemini) models. Table 6 and Table 7 show that larger models tend to exhibit higher performance across different types of logic. We further investigate an additional model with an intermediate size: Yi-34B-chat (Young et al., 2024) (results are presented in App. I). When compared to the Llama2-7B, the Yi-34B model (5x larger than Llama-7B) shows improvement in average performance across three logic types. Similarly, GPT-4 outperforms Yi-34B. This suggests that increasing the model size leads to substantial gains in performance, indicating the influence of larger model capacities on carrying out better logical reasoning.
Performance of BQA vs. MCQA From Table 6 and Table 7, we can see the overall performance of LLMs is higher on PLMCQA compared to PLBQA. Conversely, the performance is lower on NMMCQA compared to NMBQA. For PL, the performance gaps between the CT and DD inference rules primarily contributed to this trend, and
DRO, RAP, and DRD for NM. We analyze the reasoning chains associated with these inference rules and presented our detailed observations in App. I.
Effect on other logic datasets We trained the T5-large model on the LogicBench(Aug) resulting in a model named LogicT5. Furthermore, we performed fine-tuning on four other logical reasoning datasets: LogiQA, Reclor, LogicNLI, and FOLIO. Further discussion is presented in App. I.
5 Conclusions
In this work, we evaluated the logical reasoning ability of LLMs on 25 distinct inference rules and reasoning patterns covering PL, FOL, and NM logics. To this end, we introduced LogicBench, a natural language question-answering dataset focusing on evaluating a single inference rule. We devised two tasks using LogicBench: (i) BQA, and (ii) MCQA. We evaluated a range of LLMs including GPT-4, ChatGPT, Gemini-Pro, Llama-2, and Mistral on both tasks. Experimental results showed that LLMs do not perform well on LogicBench, even though they require the application of only a single inference rule. Furthermore, we also augmented LogicBench to LogicBench(Aug), which can be utilized for training purposes. Using LogicBench(Aug), we demonstrated that LLMs trained using it showcase an improved understanding of logical reasoning, resulting in a better performance on existing logic datasets.
Limitations
While LogicBench encompasses 25 distinct inference rules spanning three logic types (significantly more than any previous study) to comprehensively evaluate the logical reasoning capabilities of LLMs, it can be further extended by incorporating additional inference rules and logic types. However, with respect to first-order logic and logics with quantified variables, there can be an infinite number of such rules. In this study, we focused solely on evaluating model performance using a single inference rule; however, an interesting future direction can be enhancing the depth of reasoning complexity (i.e., multi-step reasoning) by incorporating combinations of inference rules to derive conclusions. We also note that this research is limited to the English language and can be extended to multilingual scenarios for evaluating the logical reasoning ability of LLMs.
Ethics Statement
We have used AI assistants (Grammarly and ChatGPT) to address the grammatical errors and rephrase the sentences.
Acknowledgement
We thank the anonymous reviewers for constructive suggestions, and the computer science graduate students of Arizona State University (ASU) who helped with the human annotations. We extend our gratitude to the Research Computing (RC) at ASU for providing computing resources for experiments. We acknowledge support by a 2023 Spring Amazon Research Award (ARA), and an award by Cisco via Silicon Valley Foundation.
References Jean-Philippe Bernardy and Stergios Chatzikyriakidis.
2020. Fracas: Temporal analysis.
Sajjad Beygi, Maryam Fazel-Zarandi, Alessandra Cervone, Prakash Krishnan, and Siddhartha Jonnalagadda. 2022. Logical reasoning for task oriented dialogue systems. In Proceedings of the Fifth Workshop on e-Commerce and NLP (ECNLP 5), pages 68–79, Dublin, Ireland. Association for Computational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.
Xiang Chen, Xin Xie, Ningyu Zhang, Jiahuan Yan, Shumin Deng, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2021. Adaprompt: Adaptive promptbased finetuning for relation extraction. arXiv eprints, pages arXiv–2104.
Peter Clark, Oyvind Tafjord, and Kyle Richardson. 2021. Transformers as soft reasoners over language. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 3882–3890.
Leyang Cui, Yu Wu, Jian Liu, Sen Yang, and Yue Zhang. 2021. Template-based named entity recognition using BART. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages
1835–1845, Online. Association for Computational Linguistics.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368–2378, Minneapolis, Minnesota. Association for Computational Linguistics.
Avia Efrat and Omer Levy. 2020. The turking test: Can language models understand instructions? arXiv preprint arXiv:2010.11982.
Himanshu Gupta, Neeraj Varshney, Swaroop Mishra, Kuntal Kumar Pal, Saurabh Arjun Sawant, Kevin Scaria, Siddharth Goyal, and Chitta Baral. 2023. “john is 50 years old, can his son be 65?” evaluating NLP models’ understanding of feasibility. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 407–417, Dubrovnik, Croatia. Association for Computational Linguistics.
Tanmay Gupta, Amita Kamath, Aniruddha Kembhavi, and Derek Hoiem. 2021. Towards general purpose vision systems. arXiv preprint arXiv:2104.00743.
Christopher Hahn, Frederik Schmitt, Jens U. Kreber, Markus Norman Rabe, and Bernd Finkbeiner. 2021. Teaching temporal logics to neural networks. In International Conference on Learning Representations.
Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Luke Benson, Lucy Sun, Ekaterina Zubova, Yujie Qiao, Matthew Burtell, et al. 2022. FOLIO: Natural language reasoning with first-order logic. arXiv preprint arXiv:2209.00840.
Peter Hase and Mohit Bansal. 2022. When can models learn from explanations? a formal framework for understanding the roles of explanation data. In Proceedings of the First Workshop on Learning with Natural Language Supervision, pages 29–39, Dublin, Ireland. Association for Computational Linguistics.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423–438.
Pratik Joshi, Somak Aditya, Aalok Sathe, and Monojit Choudhury. 2020. TaxiNLI: Taking a ride up the NLU hill. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 41–55, Online. Association for Computational Linguistics.
Daniel Khashabi. 2019. Reasoning-Driven QuestionAnswering for Natural Language Understanding. University of Pennsylvania.
Kirby Kuznia, Swaroop Mishra, Mihir Parmar, and Chitta Baral. 2022. Less is more: Summary of long instructions is better for program synthesis. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4532– 4552, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Teven Le Scao and Alexander M Rush. 2021. How many data points is a prompt worth? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2627–2636.
Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2012. The Winograd Schema Challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning, KR’12, pages 552–561. AAAI Press, Rome, Italy.
Vladimir Lifschitz. 1989. Benchmark problems for formal nonmonotonic reasoning: Version 2.00. In Non-Monotonic Reasoning: 2nd International Workshop Grassau, FRG, June 13–15, 1988 Proceedings 2, pages 202–219. Springer.
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, et al. 2021. Few-shot learning with multilingual language models. arXiv preprint arXiv:2112.10668.
Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2021a. Logiqa: a challenge dataset for machine reading comprehension with logical reasoning. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 3622–3628.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021b. Pretrain, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming fewshot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8086–8098, Dublin, Ireland. Association for Computational Linguistics.
Man Luo, Sharad Saxena, Swaroop Mishra, Mihir Parmar, and Chitta Baral. 2022. Biotabqa: Instruction learning for biomedical table question answering. arXiv preprint arXiv:2207.02419.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. The SICK (Sentences Involving Compositional Knowledge) dataset for relatedness and entailment.
R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2021. Metaicl: Learning to learn in context. arXiv preprint arXiv:2110.15943.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannaneh Hajishirzi. 2022a. Reframing instructional prompts to GPTk’s language. In Findings of the Association for Computational Linguistics: ACL 2022, pages 589–612, Dublin, Ireland. Association for Computational Linguistics.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2021. Cross-task generalization via natural language crowdsourcing instructions. arXiv preprint arXiv:2104.08773.
Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, Peter Clark, Chitta Baral, and Ashwin Kalyan. 2022b. NumGLUE: A suite of fundamental yet challenging mathematical reasoning tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3505–3523, Dublin, Ireland. Association for Computational Linguistics.
Swaroop Mishra and Elnaz Nouri. 2023. HELP ME THINK: A simple prompting strategy for non-experts to create customized content with models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 11834–11890, Toronto, Canada. Association for Computational Linguistics.
Mutsumi Nakamura, Santosh Mashetty, Mihir Parmar, Neeraj Varshney, and Chitta Baral. 2023. LogicAttack: Adversarial attacks for evaluating logical consistency of natural language inference. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13322–13334, Singapore. Association for Computational Linguistics.
OpenAI. 2023. Gpt-4 technical report.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.
Mihir Parmar, Swaroop Mishra, Mirali Purohit, Man Luo, Murad Mohammad, and Chitta Baral. 2022. InBoXBART: Get instructions into biomedical multitask learning. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 112–128, Seattle, United States. Association for Computational Linguistics.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080–2094, Online. Association for Computational Linguistics.
Pruthvi Patel, Swaroop Mishra, Mihir Parmar, and Chitta Baral. 2022. Is a question decomposition unit all we need? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4553–4569, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Mohammed Saeed, Naser Ahmadi, Preslav Nakov, and Paolo Papotti. 2021. RuleBERT: Teaching soft rules to pre-trained language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1460–1476, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2021. Multitask prompted training enables zero-shot task generalization. ICLR.
Abulhair Saparov and He He. 2023. Language models are greedy reasoners: A systematic formal analysis of chain-of-thought. In The Eleventh International Conference on Learning Representations.
Timo Schick and Hinrich Schütze. 2021. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 255–269, Online. Association for Computational Linguistics.
Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L. Hamilton. 2019. CLUTRR: A diagnostic benchmark for inductive reasoning from text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4506–4515, Hong Kong, China. Association for Computational Linguistics.
Oyvind Tafjord, Peter Clark, Matt Gardner, Wen-tau Yih, and Ashish Sabharwal. 2019a. Quarel: A dataset and models for answering questions about qualitative relationships. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7063– 7071.
Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. 2021. ProofWriter: Generating implications, proofs, and abductive statements over natural language. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3621–3634, Online. Association for Computational Linguistics.
Oyvind Tafjord, Matt Gardner, Kevin Lin, and Peter Clark. 2019b. QuaRTz: An open-domain dataset of qualitative relationship questions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5941–5946, Hong Kong, China. Association for Computational Linguistics.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149–4158, Minneapolis, Minnesota. Association for Computational Linguistics.
Jidong Tian, Yitian Li, Wenqing Chen, Liqiang Xiao, Hao He, and Yaohui Jin. 2021. Diagnosing the firstorder logical reasoning ability through LogicNLI. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3738–3747, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
Bertie Vidgen, Dong Nguyen, Helen Margetts, Patricia Rossini, and Rebekah Tromble. 2021. Cad: the contextual abuse dataset.
Liwen Wang, Rumei Li, Yang Yan, Yuanmeng Yan, Sirui Wang, Wei Wu, and Weiran Xu. 2022a. Instructionner: A multi-task instruction-based generative framework for few-shot ner. arXiv preprint arXiv:2203.03903.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022b. Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP
tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5085–5109, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. ICLR.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837.
Orion Weller, Nicholas Lourie, Matt Gardner, and Matthew E. Peters. 2020. Learning from task descriptions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1361–1375, Online. Association for Computational Linguistics.
Tongshuang Wu, Michael Terry, and Carrie Jun Cai. 2022. Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pages 1–22.
Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abzianidze, and Johan Bos. 2019. Can neural networks understand monotonicity reasoning? In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 31–40, Florence, Italy. Association for Computational Linguistics.
Qinyuan Ye and Xiang Ren. 2021. Learning to generate task-specific adapters from task description. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 646–653, Online. Association for Computational Linguistics.
Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019. Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3914–3923, Hong Kong, China. Association for Computational Linguistics.
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. 2024. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652.
Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. 2020. Reclor: A reading comprehension dataset requiring logical reasoning. In International Conference on Learning Representations.
Honghua Zhang, Liunian Harold Li, Tao Meng, Kai-Wei Chang, and Guy Van Den Broeck. 2023. On the paradox of learning to reason from data. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pages 3365–3373.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223.
Ruiqi Zhong, Kristy Lee, Zheng Zhang, and Dan Klein. 2021. Adapting language models for zero-shot learning by meta-tuning on dataset and prompt collections. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2856–2878, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. 2019. “going on a vacation” takes longer than “going for a walk”: A study of temporal commonsense understanding. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3363–3369, Hong Kong, China. Association for Computational Linguistics.
A Example Prompt for Sentence Generation
Below is the general prompt structure prompted to generate data. The prompt schema, as depicted in Figure 3, comprise three crucial components:
Definition provides a detailed explanation of the task and offers a natural language representation of the reasoning pattern for which we are generating sentences.
Examples provide sample sentences that need to be generated. We also illustrate how these sentences will be utilized in later stages, emphasizing the importance of coherence and the inclusion of relevant ontological concepts.
Format We provide specific formatting instructions to guide the generation of sentences.
Figure 4 illustrates an example prompt for the inference rule, namely, ‘modus tollens’ from propositional logic (PL). Modus tollens is formally represented as ((p → q) ∧ ¬q) ⊢ ¬p, which can be understood in natural language as “If p implies q, and we know ¬q, then we can conclude ¬p.” In this prompt, the definition provides a comprehensive description of the inference rule in natural language. To encourage the generation of more relevant and coherent sentences, the prompt includes an examples section that demonstrates how the generated sentences will be utilized in a later stage. This serves, as an illustration, to guide GPT-3 in producing suitable outputs. In Figure 4, we present three examples involving sentences p and q, along with their respective contexts and questions. The prompt also includes instructions on how the generated sentences should be formatted.
B Extended Related Work
As LLMs such as GPT-4, and Bard continue to evolve rapidly, it becomes increasingly crucial to
evaluate their diverse language capabilities, as well as those of forthcoming LLMs. Recently, many datasets have been created that evaluate different language understanding skills such as pronoun resolution (Sakaguchi et al., 2021; Levesque et al., 2012), commonsense reasoning (Talmor et al., 2019), numerical reasoning (Dua et al., 2019; Patel et al., 2021; Mishra et al., 2022b), qualitative reasoning (Tafjord et al., 2019b,a), temporal reasoning (Zhou et al., 2019), and feasibility reasoning (Gupta et al., 2023). Now, we present the advancements in prompt and instruction tuning using LLMs.
Prompt Learning The introduction of LLMs has significantly shifted the research trend in NLP to prompt-based learning methodologies (Liu et al., 2021b). Many studies have been conducted to investigate the efficacy of prompt-based learning in various applications including Text classification (Yin et al., 2019), Natural Language Inference
(NLI) (Schick and Schütze, 2021), and Question Answering (QA) (Jiang et al., 2020), Information Extraction (IE) (Chen et al., 2021; Cui et al., 2021), to name a few. In a recent development, the T0 model employs prompts to achieve zero-shot generalization across various NLP tasks (Sanh et al., 2021). Le Scao and Rush (2021) suggested that the use of prompts could be as valuable as hundreds of data points on average.
Instruction Learning Efrat and Levy (2020) was focused on whether existing LLMs understand instructions. The same work in the field of instruction by Hase and Bansal (2022); Ye and Ren (2021); Gupta et al. (2021); Zhong et al. (2021) has been proposed to show that models follow natural language instructions. In addition, Weller et al. (2020) developed a framework focusing on NLP systems that solve challenging new tasks based on their description. Mishra et al. (2021) have proposed natural language instructions for cross-task generalization of LLMs. Similarly, PromptSource (Sanh et al., 2021) and FLAN (Wei et al., 2021) were built for leveraging instructions and achieving zero-shot generalization on unseen tasks. Moreover, Parmar et al. (2022) shows the effectiveness of instructions in multi-task settings for the biomedical/clinical domain. Furthermore, Mishra et al. (2022a) discussed the impact of task instruction reframing. Min et al. (2021) introduced a framework to better understand in-context learning. Ouyang et al. (2022) proposed the InstructGPT model that is fine-tuned with human feedback to follow instructions. Wang et al. (2022a) has developed an instruction-based multitask framework for few-shot Named Entity Recognition (NER) tasks. In addition, many approaches have been proposed to improve model performance using instructions (Wu et al., 2022; Lin et al., 2021; Wang et al., 2022b; Luo et al., 2022; Kuznia et al., 2022; Patel et al., 2022; Mishra and Nouri, 2023).
Logic and NLI Datasets FraCas (Bernardy and Chatzikyriakidis, 2020) offers a unique approach to temporal semantics by converting syntax trees into logical formulas tailored for inference, emphasizing temporal elements such as references, adverbs, aspectual classes, and progressives. The Monotonicity Entailment Dataset (MED) (Yanaka et al., 2019) dives deep into monotonicity reasoning within NLI, probing the synergy between lexical and syntactic structures and spotlighting inherent challenges in both upward and downward monotonic reasoning trajectories. The SICK
(Marelli et al., 2014) dataset, with its foundation in 10,000 English sentence pairs, is designed to rigorously evaluate semantic relatedness and entailment, leveraging crowdsourced annotations for precision. HANS, or Heuristic Analysis for NLI Systems (McCoy et al., 2019), stands out by rigorously scrutinizing the dependability of NLI models, putting the spotlight on potential pitfalls tied to syntactic heuristics such as lexical overlap. Lastly, CAD (Vidgen et al., 2021) introduces a meticulously crafted dataset from Reddit entries, targeting the detection of online abuse. Nakamura et al. (2023) introduced LogicAttack, a method for performing adversarial attacks on NLI models using PL inference rules and proposed dataset of ∼ 9k attack samples derived from the SNLI dataset. In contrast to these works, LogicBench evaluates the logical reasoning capabilities of LLMs beyond the NLI task, focusing on the application of individual inference rules.
C Examples of Data Instances
This section provides examples of (context, question, answer) tuples corresponding to each inference rule and reasoning pattern. Additionally, it highlights the diverse range of question variations within the dataset associated with each inference rule and reasoning pattern.
C.1 Word Cloud
Figure 5 provides a word cloud derived from the LogicBench(Eval). This word cloud highlights the logical nature and diversity of our evaluation dataset. Words such as ‘if’, ‘normally’, ‘usually’, and ‘then’ are prominently featured, suggesting their frequent use in the dataset, and suggesting the logical nature of the dataset. Moreover, we can also observe several words consisting of different ontologies such as ‘cat’, ‘car’, ‘garden’, and many more, suggesting diversity in the dataset.
C.2 Propositional Logic (PL)
Here, we discuss examples of each inference rule present in the PL of the LogicBench as shown in Table 9. Table 9 has context related to the inference rule and different variations of the question according to the rule. For instance, the first row of Table 9 shows the example for inference rule, Hypothetical Syllogism (HS), formally expressed as ((p → q)) ∧ (q → r)) ⊢ (p → r). The context represents the premise, i.e., ((p → q)) ∧ (q → r)),
and the first question (Q1) represents the conclusion, i.e., p → r. Hence, Q1 is labeled as "Yes" since it supports the conclusion given the logical context. Furthermore, Q2 to Q4 represent different variations of the question by utilizing the variables (p,¬p, r,¬r). For the HS, given the provided context, Q2 to Q4 contain the variations ¬p → r, p → ¬r, and ¬p → ¬r, respectively, and are labeled as "No" since they do not support the conclusion.
C.3 First-Order Logic (FOL)
Here, we discuss examples of each inference rule and two axioms (i.e., Existential Instantiation and Universal Instantiation) present in the FOL from the LogicBench as shown in Table 10. Existential Generalization (EG), formally expressed as P (a) ⇒ ∃xP (x) indicates that there is an element a in the domain for which P (a) is true, then we know that ∃xP (x) is true. Universal Instantiation formally expressed as ∀xA ⇒ A{x 7→ a} indicates that a statement holds true for all instances (x) within a specific category A, hence it is also true for specific instance a.
Table 10 represents context related to the inference rule and variations of the question. The process of generating data instances for FOL follows a similar approach to that of PL. For example, the first row of Table 10 shows the example for axiom, Existential Instantiation (EI), formally expressed as ∃xP (x) ⇒ P (a). The context represents the initial premise ∃xP (x) and the first question (Q1) represents the conclusion, i.e., P (a). Hence, Q1 is labeled as "Yes" since it supports the conclusion given the logical context. Furthermore, we generate the only variant of the question based on ¬P (a) and labeled it as No since it does not support the conclusion.
C.4 Non-Monotonic (NM) Reasoning
Here, we discuss examples of each reasoning pattern present in the NM reasoning from the LogicBench as shown in Table 11. Table 11 has context related to the reasoning pattern and different variants of the question. For example, the first row of Table 11 shows the example for Default Reasoning with Irrelevant Information (DRI). For this reasoning, based on the given context, there are also two possible variations of the question where one with a correct conclusion labeled as Y es and another with an incorrect conclusion labeled as No.
D Examples of NL Conversion
This section illustrates the way natural language logical context and questions are created using the generated sentence in Stage 1 in addition to prompt-based templatized-context to narrative conversion. Table 13 shows examples of how context and question are generated from sentences corresponding to each inference rule for PL and FOL. Similarly, Table 14 shows examples of NM reasoning. From Table 13, we can see an example of sentence pairs (p, q) and their corresponding negation pairs (¬p,¬q) for the ‘modus tollens’ inference rule from PL. These pairs are utilized to generate templatized logical context which is converted to narrative using a specific prompt[1D] designed for PL. In addition to the narrative, the question and its variations are also created using these sentence pairs. Similarly, in the second row, we have four generic rules with variable x (p(x), q(x), r(x), s(x)) and their specific cases (i.e., x = a), along with their respective negative sentence pairs [(p(a),¬p(a)), (q(a),¬q(a)), (r(a),¬r(a)), (s(a),¬s(a))]. In the same manner, as PL, these examples demonstrate the generation of narrative and questions for the FOL inference rule called ‘Bidirectional Dilemma (BD)’, as shown in Table 13 using specific prompt[2D] designed for FOL. From Table 14, the first row presents an example of narrative and questions generated from a sentence pair for the ‘Default Reasoning with Irrelevant Information (DRI)’ from NM reasoning. In this specific instance, the generated sentences are (p, q, r, s, t), and the negation is only required for the sentence t. Therefore, there is a single negation pair (t,¬t), which is used to generate questions specific to the ‘DRI’. Same as PL and FOL, templatized context is also converted to narrative using a prompt[3D] created for NM.
Each prompt is shown below for PL, FOL, and NM templatized context to narrative conversion.
1. Prompt for templatized to narrative conversion for PL(MT):
Rule: Condition: If p then q; Situation: not q Context: Templatized context Improve the context in human-like language and make story with rephrased sentences. Instructions for generating a good story: 1. While generating a story, Use the rephrased sentences from the context of the story. 2. Make sure to include sentences corresponding to the condition and situation from the rule in the story. 3. Do not add any other extra information. 4. To generate a story, DO NOT change the name of the main character from the context, if there’s any. If the main character within the context is ’I’, please generate a story with ’I’ as the main character. 5. Generate only one paragraph with rephrased sentences.
2. Prompt for templatized to narrative conversion for FOL:
Context: Templatized context Improve the context in human-like language and make a rephrase sentences around it. Instructions for generating a good story: 1. Make sure to include only rephrased sentences in the story. 2. Do not add any other extra information and do not reach any conclusion based on context. 3. To generate a story, use only generic sentences without mentioning any main character. 4. Extract a specific case sentence from the context, rephrase it, and append it at the end of the context only. 5. Generate only one paragraph with rephrased sentences. 6. DO NOT add any prefixes.
3. Prompt for templatized to narrative conversion for NM:
Context: Templatized context Improve the context in human-like language and rephrase sentences around it. Instructions for generating a good story: 1. Make sure to include only rephrased sentences in the story. 2. Do not add any other extra information, but add some story-like context. 3. To generate a story, DO NOT change the name of the main character from the context, if there’s any. Use the same names/words for main character. If the main character within the context is ’I’, please generate a story with ’I’ as the main character. 4. Generate only one paragraph with rephrased sentences.
To ensure the quality of the narrative in the LogicBench(Eval) for task 1 and task 2, we have created category-specific prompts to convert the templatized context to convert more human-like narrative. In total, we have created three different prompts each for PL[1D], FOL[2D], and NM[3D. The prompts are designed to ensure that the logical connection is established in the narrative. Each rule of PL, FOL, and NM has a unique logical progression that should be followed in the narrative is the motivation for us to go with three different instruction-based prompts. In the prompt, used for PL rules, the logical rule is also mentioned in two parts "Condition" and "Situation". Consider the example of logical rule MT - "If p then q; not q; therefore not p", the templatized context will have "If p then q; not q;" so condition will be the first part "If p then q" while the situation will be the second part "not q". In other terms, the specific conditions are the rules to be followed while the situation is the case present in the logical rule. In contrast, for FOL, we do not need to focus more on the specific condition rather we have to make sure a generalized case is present in the narrative with 1-2 specific sentences related to the rule. For NM, we do not have such a rule instead we have a logical connection between sentences and hence we go with the instruction-based prompting.
E Task Instance Generation
This section discusses the task instance generation step, which is the last step in Fig. 2 in detail. The overall section is comprised of two subsections for BQA and MCQA. The subsection related to BQA
discusses the variation generation step in detail with question generation while the MCQA subsection provides details of prompts used for incorrect option generation along with five pre-defined sets of questions.
E.1 Task 1- BQA (variation generation)
As seen in Fig. 2, BQA has a narrative along with question, and answer pairs with variation in question and corresponding answer. As discussed in the main paper Data creation3.2, the narrative is created using the pre-defined rules for PL, FOL, and NM while a question is asked based on what can be entailed from the given context. For example, HS can be defined as "If p then q; if q then r; therefore, if p then r" and the narrative will have "If p then q; if q then r" and the question is asked as "Can we conclude if p then r?" having answer as "yes". Now, other variations in the question are asked in the following ways by negating the sentence (p and r) comb: Variation 1: "Can we conclude if ¬p then r?", Variation 2: "Can we conclude if p then ¬r?", Variation 3: "Can we conclude if ¬p then ¬r?". If there’s only one axiom (p) present in the question then there are only a 2 variations that can be made asking about (p) and (¬p).
E.2 Task 2-MCQA (Question selection and incorrect option generation)
We have MCQA as task 2 in LogicBench. In this, we have one correct option from the four options, and three options are incorrectly generated using prompting. The question is a bit different from the BQA question formation as we have MCQA and have to identify which conclusion can be derived from the narrative. The question is randomly selected from the pre-defined set of questions. Here, the five different questions are as follows:
1. What would be the most appropriate conclusion based on the given context?
2. Considering the provided context, what conclusion would be deemed most suitable?
3. In light of the context provided, what conclusion can be considered the most appropriate?
4. Based on the context, what conclusion would be deemed most suitable?
5. Taking into account the context provided, what conclusion would be most appropriate?
The narrative for MCQA is the same as the BQA’s narrative and there’s no change. Comparing BQA methodology with MCQA, MCQA’s correct option is the question asked in the BQA which can be concluded from the logical rule present in the narrative. For example, HS can be defined as "If p then q; if q then r; therefore, if p then r" and the narrative will have "If p then q; if q then r" and the question is randomly chosen from the set of questions as mentioned above and the correct option from the different multiple choice will be "if p then r". Based on the information present, we have generated an incorrect option using prompting[4E.2]. Refer this link for more examples - https://anonymous.4open. science/r/LogicBench-EEBB.
4. Prompt for incorrect option generation:
Context: Narrative Question: Randomly chosen question Correct option: Conclusion from narrative
Create three more irrelevant, incorrect options to create multiple choice. Sometime use different persons name as well in creating incorrect options. Instruction to generate options: 1. Generate only three unique choices each in new line. 2. Do not use any prefix while generating.
F Mitigation of Errors in LogicBench
While validating LogicBench(Eval), we encountered errors within the synthetically generated narrative-based context. We mitigate these errors manually, categorizing them into two groups: (i) eliminating leakage of logical conclusions and (ii) ensuring the inclusion of intended logical premises. In the first category, we found ∼ 15% of the narrative-based contexts (out of 500 total instances) were found to explicitly present the logical conclusion as a response to the question, bypassing the logical derivation process. This enables the model to extract the final logical conclusion from the context rather than derive it logically. In the second category, we found ∼ 8% of the narrative-based contexts (out of 500 total instances) where narration lacked some necessary premise sentences crucial for reaching a logical conclusion. To address this issue, we manually incorporated those sentences to ensure the quality of the data. For the MCQA task, during the generation of three incor-
rect options, we found instances where the model produces two semantically similar options, resulting in the creation of ambiguous choices in ∼ 11% of the cases (out of a total of 500 instances). We manually mitigated all these errors from the data instances ensuring the high quality of our validation data. We believe that these two versions aim to accommodate different evaluations to explore the logical reasoning capabilities of LLMs.
G Few-shot Experiments
This section discusses the performance of the different LLMs in a few-shot setting on the LogicBench(Eval)BQA. Here, we only present a case study on the BQA task. For the fair comparison with Table 6, we analyze an average performance across A(Y es). Table 15 shows the performance for each inference rule and reasoning patterns achieved by Llama-2, Mistral, Gemini, ChatGPT, and GPT-4.
As suggested in (Lu et al., 2022), prompting models are sensitive to in-context examples. Hence, we see mixed performance in Table 15 across all models. From Table 15, we can observe that incontext examples are helpful for Llama-2 since it consistently outperforms zero-shot baselines by large margins in terms of A(Y es). Llama-2 is remarkably good at following the in-context exemplars and mimicking the process to reach the correct conclusions. Thus, leveraging the in-context exemplars, Llama-2 achieves high accuracy in a few-shot setting. Specifically, Mistral consistently shows degraded performance for all logic types. However, ChatGPT improves performance on NM reasoning. Improved performance in NM reasoning demonstrates that the inclusion of in-context examples enhances the ability of these models to comprehend the nuanced meanings of logical terms such as “usually” and “typically”. In particular, we see that Gemini and GPT-4 improve performance on PL and FOL, respectively, but show competitive performance on NM.
H Experimental Setup
H.1 Extended Discussion on Experiments Zero-shot setting We evaluate GPT-4, and ChatGPT (GPT-3.5-Turbo) by utilizing their APIs provided by OpenAI3. We evaluate Google GeminiPro by utilizing its API provided by Google4 The
3https://platform.openai.com/docs/guides/gpt 4https://ai.google.dev/
evaluation is conducted on the versions of GPT-4, ChatGPT, and Gemini released in January 2024. It’s important to note that these models are regularly updated, so when reproducing the results presented in Table 6 and Table 7 (main paper), there is a possibility of variations. For Llama-2 and Mistral, we utilize the 7B-Chat, and 7B-Instruct-v0.2 versions, respectively, from the huggingface model repository5.
Experiments on other logic datasets In single and multi-task experiments on other logic datasets, we fine-tune the T5-large model for 10 epochs with a batch size of 16, 1024 maximum input length, an adaptive learning rate of 5e− 05, and an AdamW optimizer for each experiment. All experiments are performed using NVIDIA RTX A6000 GPUs.
H.2 Prompts
All the experiments conducted in the zero-shot setting were performed using three distinct prompts. The reported results in Table 6 (main paper) represent the average performance across these prompts. All the prompts follow the common pattern which includes task description and formatting instructions. The following are the three different prompts utilized in the experiments:
Prompt 1:
Given the context and question, think stepby-step logically to answer the question. Answer the question ONLY in ‘yes’ or ‘no’. Please use the below format: Context: [text with logical rules] Question: [question based on context] Reasoning steps: [generate step-by-step reasoning] Answer: Yes/No
Prompt 2:
Let’s think step-by-step to answer the question given context. Answer the question ONLY in ‘yes’ or ‘no’. Please use the below format: Context: [text with logical rules] Question: [question based on context] Reasoning steps: [generate step-by-step reasoning] Answer: Yes/No
5https://huggingface.co/models
Prompt 3:
Given the context that contains rules of logical reasoning in natural language and question, perform step-by-step reasoning to answer the question. Based on context and reasoning steps answer the question ONLY in ‘yes’ or ‘no’. Please use the below format: Context: [text with logical rules] Question: [question based on context] Reasoning steps: [generate step-by-step reasoning] Answer: Yes/No
For MCQA task, the reported results in Table 7 (main paper) represent the average performance across three prompts. All the prompts follow the common pattern which includes task description and formatting instructions. The following are the three different prompts utilized in the experiments:
Prompt 1:
Given the context that contains rules of logical reasoning in natural language, question, and options, perform step-by-step reasoning to answer the question. Answer the question ONLY in ’choice_1’ or ’choice_2’ or ’choice_3’ or ’choice_4’. DO NOT generate any text with it. Based on context and reasoning steps, choose the correct option. Please use the below format: Context: [text with logical rules] Question: [question based on context] Options: [four different choices] Reasoning steps: [generate step-by-step reasoning] Answer: choice_1 / choice_2 / choice_3 / choice_4
Prompt 2:
Let’s think step-by-step to choose the correct option for the question given context. Answer the question ONLY in ’choice_1’ or ’choice_2’ or ’choice_3’ or ’choice_4’. DO NOT generate any text with it. Based on context and reasoning steps, choose the correct option. Please use the below format:
Context: [text with logical rules] Question: [question based on context] Options: [four different choices] Reasoning steps: [generate step-by-step reasoning] Answer: choice_1 / choice_2 / choice_3 / choice_4
Prompt 3:
Given the context, question, and options, think step-by-step logically to answer the question by choosing the correct option. Answer the question ONLY in ’choice_1’ or ’choice_2’ or ’choice_3’ or ’choice_4’. DO NOT generate any text with it. Based on context and reasoning steps, choose the correct option. Please use the below format: Context: [text with logical rules] Question: [question based on context] Options: [four different choices] Reasoning steps: [generate step-by-step reasoning] Answer: choice_1 / choice_2 / choice_3 / choice_4
I Further Discussion on Results
Effect on other logic datasets Our experiments were carried out in two settings: single-task (finetuning and evaluation on one dataset) and multitask (fine-tuning on all four datasets combined, with separate evaluations for each dataset). App. H describes a detailed experimental setup. Table 12 represents the accuracy comparison between LogicT5 and baseline T5-large in both single-task and multi-task settings.
The results indicate that training LLMs on LogicBench(Aug) has a greater impact on logic datasets that primarily focus on logical reasoning, such as FOLIO and LogicNLI. Hence, we can observe that LogicT5 consistently outperforms the baseline for LogicT5 and FOLIO. However, LogiQA and ReClor encompass other forms of reasoning in addition to logical reasoning, hence, LogicT5 demonstrates competitive performance on them.
LLMs sometimes overlook contextual information In this case study, we manually examined all reasoning chains for PL and NM generated by GPT-4. The example below illustrates the observed
pattern. Specifically, in step 5 of the reasoning chain, the model deviates from the provided context and introduces unrelated information, resulting in an incorrect prediction. Although these patterns are not prevalent, we found that ∼ 1% of the total analyzed reasoning chains exhibited this pattern in PL and NM for BQA, and ∼ 0.8% for MCQA.
Rule: Modus Tollens (PL) Context: James had always been forgetful, especially when it came to wearing a sweater. his mother had repeatedly told him, ’If he forgets to wear a sweater, then he will catch a cold.’ throughout the day, he went about his activities, oblivious to the dropping temperature. hours later, he returned home, but James didn’t catch a cold. Question: Does this infer that James didn’t forget to wear a sweater? Gold Answer: Yes Reasoning steps: 1. James is known to be forgetful about wearing a sweater. 2. His mother stated that if he forgets to wear a sweater, he will catch a cold. 3. The temperature dropped, which would increase the likelihood of catching a cold if he were not wearing a sweater. 4. Despite the dropping temperature, James did not catch a cold. 5. The absence of a cold does not necessarily mean that James remembered to wear a sweater, as there could be other reasons why he did not catch a cold (e.g., he may have a strong immune system, spent little time outdoors, or was in a warm environment). 6. Therefore, the fact that James did not catch a cold does not logically infer that he remembered to wear a sweater. Answer: No
PL vs. FOL To get an indication of this we gave GPT-4 prompt - “Give twenty statements that have
‘if’ and ‘then’ in them”. From the results, we can observe that 13 of the 20 sentences in response to the prompt were FOL kind and only 7 were propositional kind. This shows that LLMs’ comprehend simple FOL sentences, thus, showing high overall accuracy on simpler FOL compared to PL. However, we interestingly observed that, as the depth increases, PL-based examples carrying a specific case/story become easier to understand compared to FOL-based generalized context.
Negations are hard to understand when embedded with logical rules. Here, Table 16 presents reasoning chains corresponding to PL and FOL supporting the effect of negation on their predictions.
Performance of BQA vs. MCQA By analyzing the reasoning chains of CT and DD for PL, we observe that LLMs often employ the elimination method in MCQA. They prioritize options to select the most appropriate conclusion, thereby improving performance. Conversely, for NM, the models typically struggle to differentiate between two options and tend to hallucinate by assuming information, resulting in lower performance. We present reasoning chains corresponding to PL and NM supporting their behavior on both BQA and MCQA tasks at https://github.com/Mihir3009/LogicBench.
Results on Yi-34B Table 17 provides results for both BQA and MCQA tasks on LogicBench.

heading: Future Work
text: In this study, we focused solely on evaluating model performance using a single inference rule; however, an interesting future direction can be enhancing the depth of reasoning complexity (i. e