Are LLMs classical or nonmonotonic reasoners? Lessons from generics (2024)

Abstract: Recent scholarship on reasoning in LLMs has supplied evidence of impressive performance and flexible adaptation to machine generated or human feedback. Nonmonotonic reasoning, crucial to human cognition for navigating the real world, remains a challenging, yet understudied task. In this work, we study nonmonotonic reasoning capabilities of seven state-ofthe-art LLMs in one abstract and one commonsense reasoning task featuring generics, such as ‘Birds fly’, and exceptions, ‘Penguins don’t fly’ (see Fig. 1). While LLMs exhibit reasoning patterns in accordance with human nonmonotonic reasoning abilities, they fail to maintain stable beliefs on truth conditions of generics at the addition of supporting examples (‘Owls fly’) or unrelated information (‘Lions have manes’). Our findings highlight pitfalls in attributing human reasoning behaviours to LLMs, as well as assessing general capabilities, while consistent reasoning remains elusive.1

heading: 1 Introduction
text: Generics are unquantified statements such as ‘Birds fly’ or ‘Tigers are striped’ (Carlson and Pelletier, 1995; Mari et al., 2013). They are generalisations about kinds even if exceptions are known (‘Penguins don’t fly’; Fig. 1). Humans typically accept generics even if the property in question is rare among the kind (‘Ticks carry the lime disease’; Brandone et al., 2012; Cimpian et al., 2010). Generics play a crucial role in human beliefs on whether an example of a kind has a given property (Pelletier and Asher, 1997). Human children master generics before they are able to reason about quantified statements (Hollander et al., 2002; Leslie and Gelman, 2012).
In defeasible or nonmonotonic reasoning (Sloman and Lagnado, 2005; Ginsberg, 1987; Koons,
1Resources available at: https://github.com/ aleidinger/nonmonotonic_reasoning_generics
2005), a hypothesis follows defeasibly from a premise, if the hypothesis is true in most normal cases in which the premise holds. Generics make for a rich test bed for testing nonmonotonic reasoning capabilities (Pelletier and Asher, 1997; Asher and Morreau, 1995). For example, given the generic ‘Birds fly’ the inference ‘Tweety, the bird, can fly’ is defeasibly valid (McCarthy, 1986; Reiter, 1988, i.a.), i.e., it is reasonable to assume ‘Tweety can fly’ even if exceptions are possible (‘Tweety is a penguin’) (Lascarides and Asher, 1991). A classical reasoner however would reject the generic ‘Birds fly’ upon learning that ‘Penguins don’t fly’.
Nonmonotonic reasoning is an integral part of human cognition (Russell, 2001), that helps us to navigate the real-world, e.g., by planning (Stenning and Van Lambalgen, 2012, Ch.5), a task that LLMs still struggle with (Valmeekam et al., 2023; Stechly et al., 2024). Nonmonotonic reasoning poses a greater challenge for LLMs than other reasoning tasks (Han et al., 2024) and hasn’t been featured prominently among natural language inference (NLI) (Gubelmann et al., 2023) or reasoning benchmarks (see §2).
The question of whether LLMs reason nonmonotonically or classically about generics and exceptions is intricately linked to desiderata of LLMs
558
as reasoners. LLMs are heralded for their ability to adapt to human or machine generated feedback (Shinn et al., 2023; Paul et al., 2023; Madaan et al., 2024; Pan et al., 2024, i.a.). At the same time, it is desired that they reason reliably when presented with invalid counterarguments, irrelevant information or user viewpoints. Sycophancy (Perez et al., 2023) of LLMs, i.e., susceptibility to be swayed by user belief, is a case in point that has been investigated in recent studies (Ranaldi and Pucci, 2023; Laban et al., 2023, i.a.).
As studies on reasoning patterns with generics remain scarce (Ralethe and Buys, 2022; Lin et al., 2020) and do not examine nonmonotonic reasoning, we address this gap by investigating the following research questions: 1) Do LLMs reason nonmonotonically or classically about generics? 2) Are LLMs sensitive to counter-evidence in the form of exceptions? 3) Do LLMs reason consistently and reliably by maintaining their response given supporting or unrelated examples? We test seven state-of-the-art LLMs for their reasoning capabilities about generics in the presence of exceptions (‘Penguins don’t fly’), as well as supporting (‘Owls fly’) and irrelevant exemplars (‘Lions have manes’). Across two datasets featuring both abstract and commonsense generics, we find that LLM behaviour mirrors human nonmonotonic reasoning patterns in the presence of exceptions (§5.1). However, most LLMs are not able to consistently maintain their agreement with generics given unrelated, or even supportive exemplars (§5.2). Our study highlights challenges in comparing LLM behaviour to human reasoning patterns as well as assessing reasoning capabilities more broadly, while consistent reasoning cannot be guaranteed. In Section 7, we present recommendations for a more holistic evaluation practice encompassing logical consistency measures.

heading: 2 Related Work

heading: 2.1 Generics in NLP
text: To date most works on generics focus on injecting commonsense knowledge or generics into LLMs (Gajbhiye et al., 2022; Liu et al., 2023a, i.a.), or training LLMs for knowledge/generic generation (Bhagavatula et al., 2023). (See AlKhamissi et al. (2022) for a review.) Bhakthavatsalam et al. (2020) construct GenericsKG, a large knowledge base of generics as an asset for downstream tasks such as Question Answering or explanation gener-
ation. Bhagavatula et al. (2023) design a pipeline for synthetic generation of generics using samples from GenericsKB as seeds. Allaway et al. (2023) in turn complement the data with exceptions and instantiations for each generic, but do not investigate nonmonotonic reasoning capabilities.
Most closely related to our work, Lin et al. (2020) find that LMs struggle to predict numerical knowledge in generics such as ‘Birds have two legs’. Ralethe and Buys (2022) find that pre-trained masked LMs falsely overgeneralise (Leslie et al., 2011) from generics (‘Ducks lay eggs’) to universally quantified statements (‘All ducks lay eggs’).

heading: 2.2 Nonmonotonic reasoning in NLP
text: Han et al. (2024) test nonmonotonic reasoning among other inductive reasoning tasks and find that only GPT-4 performs adequately. LLMs struggle to reason with contradictory information (Kazemi et al., 2024). Rudinger et al. (2020); Brahman et al. (2021); Bhagavatula et al. (2019) develop NLI tasks to test defeasible or abductive reasoning in pragmatics, while Pyatkin et al. (2023); Ziems et al. (2023); Rao et al. (2023) focus on defeasible reasoning and social norms. Parmar et al. (2024) introduce non-monotonic reasoning tasks inspired by Lifschitz (1989) as part of their LogicBench.

heading: 2.3 Consistency in reasoning
text: Most recent studies on reliability and consistency in reasoning examine sycophancy (Perez et al., 2023; Laban et al., 2023; Ranaldi and Pucci, 2023), consistency within multi-step reasoning or across sessions and users (Chen et al., 2023a; Wang et al., 2022). (See Liu et al. (2023b) for a review.)
Orthogonal to this, our work connects to studies of reasoning in the presence of unrelated or conflicting information. Shi et al. (2023) find that LLMs are easily confounded by irrelevant information in arithmetic reasoning. Across a variety of reasoning tasks, Wang et al. (2023a) find that OpenAI models struggle to maintain stable responses given irrelevant objections. Xie et al. (2023) find mixed evidence of LLMs being sensitive to information that contradicts prior knowledge, yet showing a form of ‘confirmation bias’ when presented with diverse viewpoints.

heading: 3 Tasks and datasets
text: We test nonmonotonic reasoning with generics using two datasets, featuring commonsense and
abstract generics. Both datasets contain generics (‘Birds fly’) accompanied by statements where the generic holds (‘Owls fly’) or doesn’t (‘Penguins don’t fly’). We refer to such examples as instantiations or exceptions respectively, and to both collectively as as exemplars.
As commonsense generics, we use the synthetic dataset of generics and exemplars released by Allaway et al. (2023) (henceforth referred to as GEN-comm). The dataset consists of ∼ 650 generics and ∼ 19.000 exemplars (E.g., ‘Hoes are used to plow fields or clear snow’; ‘Hoes can be used to cut grass’).2 Secondly, we construct an abstract reasoning dataset featuring generics (GEN-abs). Inspired by Han et al. (2024), we use categories (‘birds’) and examples (‘eagles’) from De Deyne et al. (2008) to construct generics of the form ‘Birds have property P’ and exemplars of the form ‘Eagles do (not) have property P’. The dataset contains 260 tuples of a generic paired with an exemplar.3
For both datasets, our goal is to prompt LLMs for their agreement with a generic in the presence of exemplars which confirm or contradict the generic. We use the following prompt template, including model-specific special tokens4 to signal a chat history between an assistant and a user.5
Example: 
 Is the following statement true: “Birds fly.” \nPlease answer yes or no. 
 yes 
 Penguins don’t fly.\nIs the following statement true: “Birds fly.”\nPlease answer yes or no. 

As a control study, we also replace the exception in the prompt (‘Penguins don’t fly’) with an instantiation (‘Owls fly’) or a random exemplar (‘Hoes can be used to cut grass’). Since generics in GEN-abs are abstract in nature, and to enable a consistent set-up across both datasets, we retain generics in GEN-comm that LLMs accepts when prompted with the first part of the above template, e.g., 
 Is the following statement true: “Birds fly.” \nPlease answer yes or no. 
.6
2See App. B for additional information on preprocessing. 3The dataset is available at: https://github.com/
aleidinger/nonmonotonic_reasoning_generics/blob/ main/data/abstract_generics.csv
4See Appendix A or https://huggingface.co/docs/ transformers/main/en/chat_templating for details.
5We also experiment with an alternative prompting template and Chain-of-Thought prompting. Since results are similar, they are included in Appendix F.
6See App. B for details and results on discarded generics.

heading: 4 Method

heading: 4.1 Models
text: We conduct our experiments on medium-sized open-weight models selected from the top of AlpacaEval7 and LMSys8 leaderboards, namely Llama-2-13b (Touvron et al., 2023), Mistral-7bInstruct-v0.2 (Jiang et al., 2023), Mixtral-8x7BInstruct-v0.1 (Jiang et al., 2024), Zephyr-7b-beta (Tunstall et al., 2023), WizardLM-13B-V1.2 (Xu et al., 2023), Starling-LM-7B-alpha (Zhu et al., 2023a), and OpenHermes-2.5-Mistral-7B (NousResearch, 2023).9

heading: 4.2 Prompting set-up
text: Since LLM behaviour can vary considerably with the phrasing of an instruction (Webson and Pavlick, 2022; Leidinger et al., 2023), we formulate three different instructions to test if an LLM agrees with a given generic: ‘Is the following statement true’, ‘Do you believe the following statement to be true’, ‘Do you believe that the following statement is accurate’. Since the optimal model reply is short and succinct, we follow the convention of HELM (Liang et al., 2023, p.161) in setting temperature to 0 for reproducibility across runs. We format every prompt using the chat template appropriate for each model, with no system prompt.4 To map LLM responses to labels disagree vs. agree, we use pattern matching and record whether a response starts with yes or no (Röttger et al., 2023). We aggregate responses for the three instructions via majority voting.

heading: 4.3 Statistical tests
text: To assess whether behaviour of LLMs is significantly different in the absence vs. presence of exemplars we resort to non-parametric statistical testing. Since our samples are paired, we use the Wilcoxon signed-rank test (Wilcoxon, 1992).

heading: 5 Results
text: We present our main results in Figure 2. Additional, accordant results are described in Appendix F.

heading: 5.1 Do LLMs reason nonmonotonically?
text: Since humans maintain their beliefs about truth conditions of generics (‘Birds fly’) in the presence of exceptions (‘Penguins do not fly’), we examine
7https://tatsu-lab.github.io/alpaca_eval/ 8https://chat.lmsys.org/?leaderboard 9See App. C for checkpoints and additional information.
whether challenging LLMs with an exception decreases their agreement to generics significantly. We find this to be the case for all models on both datasets (p = 0.01; see App. E for statistical test results). Notably, agreement rates drop to 0 for Llama-2, Mixtral, Starling and WizardLM on GEN-abs.

heading: 5.2 Do LLMs reason consistently?
text: In the presence of supporting evidence (instantiation) to a generic (‘Owls fly’), we expect LLM agreement to remain at 100%, but this is not the
case. While agreement rates remain high in numbers, they drop significantly for all models. On GEN-abs, only Mistral, OpenHermes, and WizardLM maintain agreement rates of > 90%, while agreement drops to < 10% for Mixtral.
Similarly, most LLMs are not able to disregard irrelevant random exemplars (exception/instantiation (shuffled)). Agreement rates decline steeply below 50% for Llama-2, Mistral, Mixtral and Zephyr on GEN-comm and to below 20% for Llama-2, Mixtral, Starling, WizardLM and Zephyr on GEN-abs. OpenHermes stands out as the only model that maintains agreement rates above 85% on both datasets. Notably, OpenHermes is the only model which has been trained on additional code data which has been shown to also help reasoning in natural language (Liang et al., 2023; Yang et al., 2024; Ma et al., 2023). Nevertheless, observed differences are statistically significant for all models on both datasets (App. E).

heading: 6 Analysis

heading: 6.1 How do LLMs reason about different types of generics?
text: GEN-comm contains both bare plural (BP) generics as well as indefinite singular (IS) generics (Leslie et al., 2009). (For example, ‘Sea snails have a hard shell, which protects them from predators’ (BP) and ‘A deciduous tree can be identified by its leaves’ (IS)). We did not find notable differences between LLM agreement to BP or IS generics in the presence of exemplars (see Figure 3). Aforementioned consistency failures persist for both types of generics.

heading: 6.2 Qualitative analysis
text: Generics in GEN-comm which are accepted in isolation, but are rejected in the presence of exceptions or instantiations include ‘Stimulants can be used to treat ADHD’ (Llama-2, Starling, Mixtral) or ‘A bobsleigh is driven by a single driver’ (Starling, Mistral, Mixtral, OpenHermes, WizardLM). Generics which are accepted no matter the exemplar presented in context include ‘Inflammatory diseases may be caused by an imbalance of the immune system’ (Llama-2, Starling, Mistral, OpenHermes), ‘A processor should be able to run a program’ (Starling, Mixtral, OpenHermes, WizardLM), ‘Experimental evidence is used to support or refute theories’, ‘An adventure has a beginning, middle, end’ (Starling, OpenHermes, WizardLM), and ‘Coin-
cidence is part of the human condition’ (Starling, Mistral, OpenHermes).
For GEN-abs, OpenHermes is the only LLM which maintains its agreement to a generic (‘Birds have property P’, ‘Mammals have property P’) in the presence of any instantiation or unrelated exemplar, but flips its decision and outputs disagreement in the presence of an exception. No LLM accepts any of the generics regardless of the exemplar it is paired with.

heading: 7 Discussion
text: With the advent of LLMs and reports of impressive performance, including on reasoning tasks (Wei et al., 2022; Kojima et al., 2022), recent investigations into failure modes in reasoning have focused, e.g., on prompt attacks (Zhu et al., 2023b; Wang et al., 2023b, i.a.), sycophancy (Perez et al., 2023; Laban et al., 2023; Ranaldi and Pucci, 2023, i.a.) or adaptability to critique or feedback (Madaan et al., 2024; Chen et al., 2023b; Huang et al., 2023; Pan et al., 2024). Such research trends might be seen as emblematic of a view of LLMs as artificial natural artifacts (Kambhampati, 2022). Results in this study demonstrate the difficulties of making claims about reasoning capabilities of LLMs or comparing them to human reasoners (Han et al., 2024; Ralethe
and Buys, 2022; Lin et al., 2020), while consistent reasoning remains elusive even for state-of-the-art LLMs. Research that predates the paradigm shift to few-shot prompting, has advocated for arguably simpler, systematic diagnostic tests (Ribeiro et al., 2020; Ettinger, 2020; Kassner and Schütze, 2020). We argue that such behavioural tests merit a revival, so that performance metrics for reasoning are complemented with measures of logical consistency and robustness.

heading: 8 Conclusion
text: The present study focuses on nonmonotonic reasoning capabilities of LLMs in the context of generics. We evaluate seven state-of-the-art LLMs on two datasets featuring both abstract and commonsense generic statements. While LLM behaviour on generics paired with exceptions is in line with nonmonotonic reasoning patterns, LLMs fail to reason consistently and robustly when adding supporting or unrelated exemplars.

heading: 9 Limitations
text: We acknowledge that our experiments exclusively feature generics and exemplars in English. Future research might profit from including additional
languages to examine nonmonotonic reasoning capabilities in other languages, drawing on crosslinguistic research on generics (Mari et al., 2013). Such work might also highlight differences in consistency failures between different languages. In this work, we do not experiment with generics pertaining to demographic groups or nationalities because of concerns around social bias. Future work might examine LLM behaviour on generic statements for larger LLMs or closed-source models. We restrict ourselves to medium-sized open-weight LLMs, due to their widespread use and availability, as well as restrictions on our computational budget.

heading: Acknowledgements
text: We thank our anonymous reviewers for their insightful comments. The work for this publication is financially supported by the project, ‘From Learning to Meaning: A new approach to Generic Sentences and Implicit Biases’ (project number 406.18.TW.007) of the research programme SGW Open Competition, which is (partly) financed by the Dutch Research Council (NWO).

heading: A Example input
text: Example input formatted using our prompt template and the chat template10, e.g., for Mistral7B-Instruct:
10https://huggingface.co/blog/chat-templates
Model # samples Mistral-7B-Instruct 2093 Llama-2-13b 1245 Zephyr-7b-beta 1536 WizardLM-13B-V1.2 2225 OpenHermes-2.5-Mistral-7B 2153 Starling-LM-7B-alpha 2244 Mixtral-8x7B-Instruct-v0.1 1959

heading: B Additional information on data preprocessing
text: For GEN-comm, we conduct additional processing to obtain high quality generics and ensure a parallel experimental setup between GEN-comm and GEN-abs. We retain only generics that were annotated as ‘valid’ by human annotators. We filter generics for which both an exception and an instantiation exists. Since generics are unquantified statements, we remove any quantifiers such as ‘generally’, ‘usually’ and ‘typically’ at the beginning of each generic. To enable consistent evaluation on GEN-abs and GEN-comm, we evaluate each LLM on generics contained in GEN-comm which it accepts a priori. In an initial experiment, we prompt LLMs using the first part of our template (above; App. A). An example input for GEN-comm would be, e.g., ‘
 Do you believe that the following statement is accurate: ‘Birds have property P.’ Please answer yes or no
’. Generics for which an LLM does not generate yes as a response are discarded. We retain > 1200 samples for each model (See Table 1 for details).
Results on the resultant dataset are presented in the main body of the paper (Section 5). For the reader’s interest, we include here also LLM re-
sponses to generics contained in GEN-comm which are rejected by LLMs, i.e., a given LLM generates the response no to the prompt above (See Figure 4). As expected agreement rates soar for almost all models when adding an instantiation which confirms the previously rejected generic. Nevertheless, agreement rates also increase, albeit less, when adding exceptions or unrelated random examplars, particularly for Llama-2 and WizardLM. OpenHermes and Starling show the least inconsistencies.

heading: C Additional information on LLMs
text: In this section we provide additional details on the models used in this study which are listed in Section 4.1. The specific checkpoints we use can be seen in Table 2 and are all available through the HuggingFace Hub. All models we use are trained for chat interaction.
Mixtral-8x7B-Instruct-v0.1 (MistralAI, 2023) is a sparse mixture of expert model based on 8 Mistral 7B models that has been further trained using supervised finetuning and Direct Preference Optimisation. It ranks highest among its weight class on AlpacaEval11 and chat.lmsys12 leaderboards (as of Feb 6 2024). At its release it surpasses GPT-3.5 and LLaMA-2-70b.
11https://tatsu-lab.github.io/alpaca_eval/ 12https://chat.lmsys.org/?leaderboard
StarlingLM-13B-V1.2 (Zhu et al., 2023a) has been trained via Reinforcement Learning from AI Feedback (RLAIF) on the Nectar dataset. In its weight class, it is the second best performing model on chat.lmsys and 4th on AlpacaEval (as of Feb 6 2024).
Amidst mounting evidence that training on code enhances reasoning abilities also for natural language (Liang et al., 2023; Yang et al., 2024; Ma et al., 2023), we also use OpenHermes-2.5-Mistral7B (NousResearch, 2023) which ranks third in its weight class on chat.lmsys. It is Mistral-based model that has been finetuned on additional code datasets. Notably, the developers detail that this results in improvements on non-code tasks.13
WizardLM-13B-V1.2 (Xu et al., 2023) is a finetuned version of Llama-2 13b and is ranked 8th in its weight-class on both chat.lmsys and AlpacaEval.
Zephyr-7b-beta (Tunstall et al., 2023) is a finetuned version of Mistral-7B-v0.1. It is ranked 9th on chat.lmsys and 11th on AlpacaEval.

heading: D Average runtime
text: Generating LLM responses for one LLM and all generics across all settings took less than 0.5 GPU hours. All experiments were conducted on one NVIDIA A100 GPU.

heading: E Statistical test results
text: Responses in the presence of exemplars are significantly different from results obtained without examplars (see Tables 3, 4, 5), for all types of exemplars and all models (significance level 0.01; sole exception is Llama-2 with CoT prompting as can be seen in Table 5 rows 1-2).
13https://huggingface.co/teknium/OpenHermes-2. 5-Mistral-7B

heading: F Additional experimental results
text: We demonstrate additional experimental results based on an alternative prompting set-up in Figures 5 and 6.
To this end, we prompt LLMs using the following template where 
 is an example of a model-specific special token used in chat templating. For example:
Prompt 
 Do you believe that the following statement is accurate: ‘Birds fly’
Please answer yes or no. 

For GEN-comm, we retain all generics to which an LLM responds yes to the prompt above. We then prompt LLMs anew supplying an exception, instantiation or random exemplar together with a generic for both datasets. For example:
Prompt 
 Penguins do not fly.
Do you believe that the following statement is accurate: ‘Birds fly’
Please answer yes or no. 

We find that results differ significantly between
the two conditions (no exemplar vs. with an exemplar) (see Table 4 for statistical test results). On GEN-comm (Figure 5) agreement rates drop considerably in the presence of exceptions which mirrors nonmonotonic reasoning patterns. Agreement is higher, yet still drops significantly in the presence of instantiations. No LLM maintains perfectly consistent responses at the addition of random instantiations or exceptions. When prompting with random exemplars surprisingly agreement drops, most notably for Llama-2 and Zephyr.
For the reader’s interest, we also include results on the portion of generics in GEN-comm which is rejected by LLMs a priori (Table 7). As expected, agreement increases from zero at the addition of an instantiation to the prompt, most notably for OpenHermes and Starling. However, LLMs should maintain a response of no at the addition of an exception or random exemplar to the prompt. This is visibly not the case with agreement rates increasing significantly for all models.
On GEN-abs, agreement drops considerably at the addition of an exception for all models except OpenHermes (Figure 6). Notably OpenHermes and Starling-LM appear to yield consistent responses in the presence of our controls, the random exemplars, while Llama-2 and Zephyr perform worst in that regard.
F.1 Chain-of-thought prompting Additionally, we ran experiments using zero-shot Chain-of-Thought (CoT) prompting in the style of (Kojima et al., 2022) by appending ‘Let’s think step by step’ to our prompts. We present results on GEN-comm in Figure 8 and results on GEN-abs in Figure 9.
On GEN-comm, agreement rates drop significantly for all models at the addition of exceptions, instantiations or shuffled exemplars (with the exception of Llama-2 when we include instantiations; see Table 5 for significance results). Agreement rates drop more given exceptions in comparison to instantiations or unrelated examplars for Mistral, Mixtral, OpenHermes and Starling. For Llama-2 and Zephyr agreement rates fall below 10% at the addition of unrelated exemplars.
On GEN-abs, agreement rates fall drastically given exceptions and equal 0% for Llama-2, Mixtral, Starling and Zephyr. The same is true for shuffled instantiations. OpenHermes is the only model to maintain agreement rates above 90% when presented with instantiations or shuffled exceptions.

heading: Future Work
text: Future research might profit from including additional
languages to examine nonmonotonic reasoning capabilities in other languages, drawing on crosslinguistic research on generics (Mari et al. , 2013). Such work might also highlight differences in consistency failures between different languages. In this work, we do not experiment with generics pertaining to demographic groups or nationalities because of concerns around social bias. Future work might examine LLM behaviour on generic statements for larger LLMs or closed-source models. We restrict ourselves to medium-sized open-weight LLMs, due to their widespread use and availability, as well as restrictions on our computational budget