Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMs (2024)

Abstract: Large language models (LLMs) have achieved impressive human-like performance across various reasoning tasks. However, their mastery of underlying inferential rules still falls short of human capabilities. To investigate this, we propose a logic scaffolding inferential rule generation framework, to construct an inferential rule base, ULogic, comprising both primitive and compositional rules across five domains. Our analysis of GPT-series models over a rule subset reveals significant gaps in LLMs’ logic understanding compared to human performance, especially in compositional and structural complex rules with certain bias patterns. We further distill these rules into a smaller-scale inference engine for flexible rule generation and enhancing downstream reasoning. Through a multijudger evaluation, our inference engine proves effective in generating accurate, complex and abstract conclusions and premises, and improve various commonsense reasoning tasks. Overall, our work sheds light on LLMs’ limitations in grasping inferential rule and suggests ways to enhance their logical reasoning abilities 1.

heading: 1 Introduction
text: “Did Leonardo da Vinci ever use a laptop for drawing pictures?” Large language models can swiftly and confidently respond “No" (Geva et al., 2021; Wang et al., 2023), demonstrating impressive reasoning ability that rivals human (OpenAI, 2023; Ouyang et al., 2022). However, when posed with more obscure questions, such as question Q2 in Figure 1, LLMs are prone to exhibit uncertainty and errors. This inconsistency raises concerns about whether LLMs grasp the underlying logic of matters as proficiently as humans (Wason, 1968) (see “underlying logic" in Figure 1) and highlights chal-
* Corresponding author 1Code and data are available at https://github.com/
SiyuanWangw/ULogic.
lenging reasoning situations (like Q2) where current LLMs might struggle.
Humans naturally abstract underlying logic (e.g., inferential rules) from extensive real-world observations (Barwise, 1993), which is beneficial for addressing diverse reasoning situations. An inferential rule is typically defined as a premise with a set of facts (e.g., “Person X died before ... earlier than B”) leading to a conclusion (e.g., ‘‘Person X cannot access Object Y”) (Boghossian, 2014). Grasping this rule enables the deduction that a person cannot access an object invented posthumously. This work utilizes symbolic logic as a scaffold to generate challenging reasoning situations for GPTseries LLMs, as shown in Figure 2. We observe a discernible gap between LLMs and humans in understanding inferential rules, especially rules with complex premises.
However, collecting such inferential rules at scale presents a major challenge. Previous work mainly relies on manual curation (Sap et al., 2019a; Sinha et al., 2019) or inductive logic programming (Qu et al., 2020), which are either laborintensive or limited in diversity. Besides, manually crafted rules often appear simple and overly specified, struggling to move beyond basic intuition or
7523
generalize across diverse situations. For example, the rule If X runs out of steam, then X becomes tired from Sap et al. (2019a) has only one premise fact and narrowly specifies exhaustion.
To this end, we introduce Logic scaffOlding Inferential Rule gEneration (LOIRE), a framework to generate inferential rules of different complexities. LOIRE operates in two stages: primitive rule generation and rule composition. Initially, we define “primitive rules” to describe abstract objects like Person and Food, and ensure they cannot be decomposed into simpler rules, facilitating broad generalization and easy generation. We then incorporate GPT-4’s generative capability and human expertise to generate primitive rules with high confidence. This process, consistently guided by symbolic logic, involves GPT-4 drafting potential conclusions in various domains, and forming premises with one or more facts. We ensure rules’ logical soundness through the model’s self-critique and human manual verification. In the second stage, we apply backward chaining (Gallaire and Minker, 2012; Al-Ajlan, 2015) upon these primitive logical rules to automatically construct compositional rules of varied lengths and structures at scale.
Using this framework, we construct ULogic, an inferential rule base with around 8, 000 primitive rules and over 6, 000 compositional rules. These rules span five key domains: object affordance, accessibility, interaction, location, and person’s need. We hope ULogic will serve as a valuable resource, facilitating the assessment of LLMs’ proficiency in underlying logic and enhancing flexible rule generation and downstream reasoning. We utilize ULogic to create an entailment probing task with a comprehensive and robust evaluation strategy, to assess LLMs’ grasp of inferential rules against human performance. Our analysis
of GPT-series LLMs, including GPT-4, GPT-3.5Turbo and GPT-3.5-Turbo-Instruct, indicates they have a basic understanding of inferential rules but fall short of human proficiency, especially in rules with complex premises. Specifically, all models struggle more as the compositional complexity increases. While GPT-4 performs consistently on verbalized and symbolic rules, the other models sharply degrade on symbolic rules. Additionally, all models exhibit disparities on various rule structures with Disjunctive-Transitive rules posing the greatest challenges. Moreover, these LLMs display notable polarity biases with GPT-4 showing a necessary bias, underscoring areas for improvement.
We further distill crafted inferential rules into a smaller-scale inference engine for flexible rule generation and downstream reasoning. We design three tasks: conclusion generation, premise completion and premise generation, to construct an instructiontuning dataset for inferential rule distillation. Experimental results through a multi-judger evaluation mechanism incorporating automatic metrics, LLM evaluators and human preferences show that our inference engine possesses the ability for these three tasks. It outperforms GPT-3.5-Turbo across all dimensions of three tasks and even surpasses GPT-4 in generating more complex and abstract rules. Moreover, it can generate logical rules that enhance downstream commonsense reasoning.

heading: 2 Logic Scaffolding for Inferential Rule Generation

heading: 2.1 Preliminary of Inferential Rules
text: To better control the generative capability of LLMs for rule generation, we focus on if-then inferential rules with variables, that can be easily expressed as symbolic logic (Novák and Lehmke, 2006). An inferential rule describes a logical implication from

heading: Primitive Rules
text: a premise (a set of facts) to a conclusion (a specific fact), where each fact is a predicate expression with two variables, and each variable has a designated variable type. For each rule, we employ logic scaffolding which first generates its symbolic expression to consistently guide its verbalized form.
We utilize Prolog (Apt et al., 1997) to formulate symbolic rules as Conclusion:-Premise, where :- indicates the logical implication. For example,
CanNotEat(Person X,Food Y ):-
AllergicTo(Person X, Substance Z),
Contains(Food Y, Substance Z). (1)
The left-hand side is the conclusion and the right hand lists premise facts connected by commas. “CanNotEat”, “AllergicTo” and “Contains” are predicate verbs while Person, Food, Substance are variable types of variables (X, Y, Z). This symbolic rule can be verbalized as: If Person X is allergic to Substance Z and Food Y contains Substance Z, then Person X cannot eat Food Y.
Primitive Rule We aim to generate primitive rules for further compositions and potential generalization. We formally define primitive rules as follows: (1) they concern abstract objects, like Person and Food, rather than specific instances, and their common properties; (2) they cannot be decomposed into simpler rules. Inspired by superordinate objects such as instrument, fruit, tool from Rosch and Mervis (1975), we assemble a collection of abstract objects. We first identify the most common tail nodes of “IsA” relations from ConceptNet (Speer et al., 2017). For those nodes that are still fine-grained, we further seek their general hypernyms by searching ConceptNet and WordNet (Miller, 1995). We totally gather a list of 32
most common abstract objects for primitive rule generation, with 18 common properties generated by prompting GPT-4, as detailed in Appendix A.1.

heading: 2.2 Primitive Rule Generation Pipeline
text: The pipeline of primitive rule generation is illustrated in Figure 3, consisting of five steps. First, we randomly select two abstract objects, and generate potential predicates between them to form conclusions. GPT-4 is prompted to generate corresponding feasible premises with both single and multiple facts, thereby constructing candidate primitive rules. We then apply heuristic methods to filter invalid and non-primitive rules, and utilize GPT-4 to select the rules it deems logically correct. We further diversify rule predicates via backward/forward chaining (Urbani et al., 2011; Shindo et al., 2021) with generated single-fact rules, and filter excessively repetitive rules. Finally, the diversified rules undergo manual verification to ensure the final set of high-confidence primitive rules.
Step-1: Conclusion Preparation From the set of abstract objects, we select any two, e.g., Person and Food, and prompt GPT-4 to generate potential predicates connecting them as conclusions, e.g., CanEat(Person X, Food Y). We attempt every possible pairing of two, where the selected objects can be identical. For each pair of objects, {object1} and {object2}, we aim to generate conclusions across five domains: {object affordance, accessibility, interaction, location and person’s need}, thereby covering diverse scenarios. Explanations and example rules of these domains are listed in Appendix A.2. The prompt for conclusion preparation about affordance is below. Besides, we negate the generated predicates to yield both positive and negative conclusions, e.g., CanNotEat(Person X, Food Y),
across object affordance, accessibility, and interaction domains, building a complete rule set.
Prompt for Conclusion Preparation
According to commonsense knowledge in reality, please list 5 predicates between the given two objects to describe the {object affordance}. Examples: Object: Show, Artwork Predicate: CanBeAdaptedFrom(Show X, Artwork Y)
Object: {object1}, {object2} Predicate:
Step-2: Premise Generation Guided by a symbolic conclusion, we prompt GPT-4 to generate its premises in both symbolic and verbalized forms for better controllability. This process involves the logit bias setting, motivating premises to describe relationships between abstract objects and their properties. Specifically, premises are generated under the constraint of logit bias, increasing the likelihood of these objects and properties appearing in the output. For each conclusion, we create both single-fact and multi-fact premises to yield candidate rules of varying lengths. We tailor instructions and demonstrations for each domain to prompt GPT-4 for premise generation exploring different possibilities, as detailed in Appendix A.3.
Step-3: Rule Filtering After over-generating candidate primitive rules, we first design several heuristic methods to filter grammatically invalid or nonprimitive rules, based on their symbolic forms. For grammatically validity, we check whether the variables in premises form a connected graph from node “X” to node “Y”, as in Appendix A.4. Regarding primitiveness, we exclude rules with nonprimitive variable types or those comprising more than 3 premise facts. Besides, we eliminate trivial rules that both contain negative words in their premise and conclusion, e.g., CanNotEat(Person X, Food Y):- CanNotAccess(Person X, Food Z).
Directly generating logically correct rules is challenge. Thus we further adopt a self-critic strategy (Gou et al., 2023) where GPT-4 critiques the accuracy of its self-generated rules in a verbalized format, and provides explanations of its judgments. When prompting GPT-4, we include two demonstrations featuring both correct and incorrect rules to mitigate label bias. These demonstrations vary across different domains. An example prompt for object affordance is in Appendix A.5.
Step-4: Rule Diversifying To increase the va-
riety of rule expressions, we diversify predicates while maintaining its logical accuracy. Based on symbolic rules, we respectively apply forward and backward chaining algorithms to their conclusion and premise with generated single-fact rules, as shown in Figure 4. In forward chaining, we take the conclusion as a new premise to generate an intermediate single-fact rule, subsequently substituting the original conclusion with this newly derived conclusion. In backward chaining, a premise is taken as a conclusion to create an intermediate single-fact rule, and replace the original premise with the new-generated one. Intermediate singlefact rules are also generated through Step-2 and 3. Each original rule undergoes one forward and one backward chaining to derive two diversified rules.
Step-5: Human Verification To obtain more reliable inferential rules, we utilize Amazon Mechanical Turk (AMT) to recruit human annotators for manual verification. For each rule, three annotators are asked to assess the clarity and comprehensibility of its premise and conclusion, as well as the logical entailment from the premise to the conclusion. Only the rules unanimously validated by all three annotators are preserved. The AMT template for human verification and the overall rates of rule acceptance are listed in Appendix A.6.

heading: 2.3 Rule Composition
text: We create more compositional rules by applying backward chaining upon primitive rules with different chaining steps. In each step, we select a premise fact from the current rule as a conclusion, deriving a new primitive rule that describes its multi-fact premise. This selected fact is then replaced with the newly generate premise. This process is iteratively conducted 1 to 3 times, creating rules with varying compositional levels (1 to 3). An example of one backward chaining step is shown in Figure 5. The intermediate primitive rules used in backward
chaining are generated via the pipeline described in Sec. 2.2, thus also contributing to our primitive rule set. As the composition of logically correct sub-rules is also logically correct, there is no need to verify these compositional rules separately.

heading: 2.4 Rule Statistics
text: Using Logic-RGC framework, we construct an inferential rule base ULogic comprising 14, 647 rules, with 7, 967 primitive and 6, 680 compositional ones. These rules span five key domains: object affordance, accessibility, interaction, location and person’s need. They vary in compositional depth from 0 to 3, with rule lengths ranging from 1 to 6. Detailed statistics are in Table 1.

heading: 3 Assessing LLMs’ Proficiency in
text: Capturing Inferential Rules
We utilize ULogic for a systematic evaluation of LLMs’ proficiency in underlying logic compared to human competence. Specifically, we select a highquality probing subset with 1,104 diverse rules from our rule base 2, and create a binary entailment
2These high-quality probing rules are verified by authors, covering various lengths, polarities and structures.
classification task for assessing whether LLMs capture the entailment within inferential rules.

heading: 3.1 Analysis Setup
text: Considering LLMs’ sensitivity to various input formulations and shortcut biases, we design a comprehensive and robust assessment mechanism to ensure reliable analysis. For each inferential rule, we convert it into five distinct probing questions to mitigate template bias, as summarized in Appendix B.1. We report the average accuracy and variance (the error line of each bar) across five templates. Besides, we adopt a two-shot chain of thought (CoT) prompting strategy (Wei et al., 2022) requiring the model to generate a rationale after presenting its answer, using ”and also explain why.” We include one correct rule and one incorrect rule in the two demonstrations to minimize label bias.
Following the Law of Non-Contradiction (Priest et al., 2006), the propositions "If X, then Y" and "If X, then not Y" are mutually exclusive that cannot both be true at the same time. To enhance the reliability of our probing, we flip each rule by negating its conclusion, and simultaneously probe both the original rule and its flipped version. A rule is accurately classified only if the original rule is affirmed (True/Right/Yes) and its flipped counterpart is negated (False/Wrong/No), as shown below. A specific example is in Appendix B.2. This dualsided probing is applied to both human and LLMs.
If Premise, then Conclusion_original. True/Right/Yes If Premise, then Conclusion_flipped. False/Wrong/No

heading: 3.2 Empirical Analysis
text: We conduct analysis on GPT-series LLMs, including GPT-4, GPT-3.5-Turbo and GPT-3.5-TurboInstruct3, aiming to investigate LLMs’ proficiency of inferential rules against human performance by exploring the following questions. The human performance is obtained by asking AMT annotators whether the input rule is logical correct with high probability. Each performance presented in following bar charts is calculated based on 150 instances randomly sampled from our probing subset. (1) How does model performance vary with increasing compositional complexity? We conduct rule probing in terms of different compostional lengths, as illustrated in Figure 6a. “Length=1,2,3,4” respectively denote rules with
3Substituting the now-deprecated Text-davinci-003.
Transitive Disjunctive Transitive-Disjunctive 0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
A cc
ur ac
y
Verbalized Rules
Human GPT-4 GPT-3.5-turbo GPT-3.5-Instruct
Figure 7: Result of varied structures.
1∼4 facts in their premises. The analysis of different compostional depths is also provided in Appendix B.3. They both reveal that as compositional complexity increases, the performance of both human and all models drops. The primary reason is that compositional complex rules typically necessitate the aggregation of multi-step reasoning, which escalates higher-order relationships understanding and exponential error accumulation with each additional step (Dziri et al., 2023). Besides, there is a persistent performance gap between all models and human, particularly pronounced with compositional complex rules, suggesting significant potential for enhancement in this area.
(2) Are LLMs proficient in capturing both symbolic and verbalized rules? We further analyze the performance of LLMs on symbolic rules (see Figure 6b), and compared it to the verbalized result in Figure 12. We observe that GPT-4 achieves consistent performance on verbalized and symbolic rules, whereas GPT-3.5-Turbo and GPT-3.5Instruct sharply degrade on symbolic rules. This suggests that the GPT-3.5 series may have limitations in generalizing across varied types of linguistic structures beyond natural language, whereas GPT-4 likely have undergone specific optimizations for symbolic interpretations.
(3) Are there performance disparities among models concerning different rule structures? Our generated multi-fact rules (Length > 1) have three intrinsic structures: Transitive, Disjunctive and Disjunctive-Transitive. Specific illustrations and examples of each structure are detailed in Appendix B.4. Figure 7 shows that DisjunctiveTransitive rules pose greater challenges compared to Transitive and Disjunctive ones, especially for GPT-3.5-Turbo and GPT-3.5-Instruct. We hypothesize that this discrepancy stems from increased compositional complexity and LLMs’ insufficient
learning of logical structures in natural language.
(4) Do LLMs exhibit a polarity bias over inferential rules? Our inferential rules contain both positive and negative conclusions. We conduct a comparative analysis of polarity discrepancy, as shown in Figure 8a. GPT-4 and GPT-3.5-Instruct exhibit a pronounced positive bias, performing better on rules with positive conclusions. This bias may originate from the imbalanced distribution of LLMs’ training data (Garg et al., 2022), with a higher proportion of positive statements. We further explore different CoT strategies with GPT-4: (1) first answer then explain (Answer-Explain), (2) first think then answer (Think-Answer), (3) self-consistently think then answer (Self-Consistency) (Wang et al., 2022). Various CoT prompts are listed in Appendix B.5. Figure 8b shows that although advanced CoT strategies can mitigate the positive bias, they adversely impact the performance on rules with both positive and negative conclusions.
(5) Why does GPT-4 significantly underperform GPT-3.5-Turbo on transitive rules? Previous analysis shows that GPT-4 generally outperforms or equals the other models, but this superiority disappears on transitive rules, as evidenced in Figure 7. We investigate this question in Appendix B.6, which reveals that GPT-4 exhibits a “necessary bias” that tend to consider all necessary conditions reaching a conclusion, avoiding definite judgement. This
conservative style may come from LLMs’ preference alignment during Reinforcement Learning with Human Feedback (Ouyang et al., 2022).
Overall, GPT-4 performs best in grasping inferential rules. But compared to human performance, there still remains substantial room for improvement across all models, especially in highly compositional, symbolic and structural complex rules. Besides, all models tend to exhibit a polarity bias towards rules with positive conclusions with GPT4 also showing a necessary bias. These findings suggest potential areas for future enhancements.

heading: 4 Rule Distillation as Inference Engine

heading: 4.1 Instruction Dataset & Model Tuning
text: For flexible rule generation and benefiting downstream reasoning, we distill our crafted rules into a smaller-scale inference engine as illustrated in Appendix C.1. We tailor three tasks: conclusion generation, premise completion and premise generation, to construct an instruction-tuning dataset for inferential rule distillation. The detailed definitions of these tasks are also described in Appendix C.1.
We gather all primitive rules and partial compositional rules to formulate the instruction-tuning dataset, as compositional rules are constructed from primitive ones. We take 10,703 rules for training and 943 for testing. Altogether, we create 39,887 instances for instruction tuning, including 10,703, 18,500 and 10,684 for conclusion generation, premise completion and premise generation. We have 3,500 testing instances, divided as 943, 1,614 and 943 for these three tasks. We use Mistral7b (Jiang et al., 2023) as the backbone model and fine-tune it with our constructed instruction dataset as our inference engine. The training details and demo page can be found in Appendix C.2.

heading: 4.2 Rule Generation Evaluation
text: We compare our inference engine against GPT-4 and GPT-3.5-Turbo across three tasks to assess rule generation. For a fair comparison, we prompt GPT4 and GPT-3.5-Turbo to simultaneously generate symbolic and verbalized responses, using similar prompts as in Step-2 of Sec. 2.2. Detailed prompts are in Appendix C.3. We introduce a multi-judger evaluation mechanism, incorporating automatic metrics, LLM evaluator and human preference. We evaluate the logical accuracy for conclusion generation and premise completion. For premise generation task with a specified number of facts, we gen-
erate three potential premises for each conclusion, and compare these premises in accuracy, diversity, complexity and abstractness. Detailed definition of these metrics are described in Appendix C.4. Automatic Evaluation For automatic accuracy evaluation of three tasks, we calculate BLEU score (Papineni et al., 2002) against reference responses. For complexity of premise generation, we assess the average fact number of three generated premises. For diversity, we compute average SelfBLEU (Shu et al., 2019; Tevet and Berant, 2020) between three generated premises. Specifically, Self-BLEU measures the BLEU score of a generated premise against another, and a high average Self-BLEU indicates low diversity. Abstractness is not easy to evaluate automatically, so we leave it to LLM evaluation. The results are shown in Table 2.
LLM Evaluation We adopt GPT-4 as an evaluator to rate the generated responses on a scale from 1 to 3. The criteria of each rating along with examples are provided to the evaluator. Please see Appendix C.5 for detailed prompts. For each task, we select 100 instances for LLM evaluation, ensuring a balance across all domains and all types (including single-fact, multi-fact, intermediate, composition=1∼3 rules) as detailed in Table 1. The rating results are presented in Table 3.
Human Evaluation To better assess premise generation in line with human value, we further recruit two annotators for each instance to compare their accuracy. We implement a pairwise comparison setting, asking annotators to determine which group
of generated premise is more accurate in terms of logical consistency with the given conclusion, commonsense alignment and correctness of fact numbers. The results are shown in Fiure 9.
From all evaluation, we can see that our inference engine enables the smaller-scale LLM with the capability for conclusion generation, premise completion and premise generation. It performs better than GPT-3.5-Turbo across all metrics in three tasks, and even outperforms GPT-4 to generate more complex and abstract rules.

heading: 4.3 Downstream Reasoning Evaluation
text: We further analyze the effectiveness of our inference engine in generating logical rules or explanations to enhance downstream reasoning tasks. We evaluate on following commonsense reasoning datasets: StrategyQA (Geva et al., 2021), SOCIAL IQA (Sap et al., 2019b), LINK (Li et al., 2023), PIQA (Bisk et al., 2020) and CSQA2.0 (Talmor et al., 2022). We use a zero-shot CoT strategy to prompt two baseline models, Mistral-7B-Instructv0.1 and Llama-2-7b-chat (Touvron et al., 2023), to answer questions with following explanations. We then utilize our inference engine to generate logical rules or explanations relevant to answer questions, and supplement these generated rationals to baseline models as input to enhance their performance. We compare the prediction accuracy of our inference engine augmented models against baselines.
The comparative results are shown in Tabel 4. Our inference engine can generate logical rules or explanations that benefit multiple downstream commonsense reasoning tasks on top of different backbone models. For the lack of clear advantage on PIQA and performance decline on CSQA2.0, we speculate that PIQA may be contaminated during Mistral’s training process, and CSQA2.0’s focus is mainly on longtail commonsense knowledge rather than requiring logical rules inference, like "Is cotton candy sometimes made out of cotton?"

heading: 5 Related Work
text: Logical Rule Generation Logical inferential rules are crucial for everyday reasoning (Geva et al., 2021; Talmor et al., 2022), and collecting these inferential rules is challenging. Prior work mainly adopts inductive logic programming (ILP) (Yang and Song, 2019; Qu et al., 2020; Sen et al., 2022) for rule generation. However, they can only generate rules from existing knowledge graphs and the generated rules has potential inaccuracies. Alternatively, Sinha et al. (2019) manually create a set of inferential rules for inductive reasoning, but their scope is limited to kinship. Sap et al. (2019a) construct a commonsense inferential rule base through crowdsourcing, but these rules tend to be overly simple and specific, struggling to move beyond basic intuition and generalize to varied situations. Abstract and complex rules are essential in tackling diverse complex questions, paving the way for complex reasoning and decision-making. Although LLMs have opened new avenues for generating inferential rules (Zhu et al., 2023), they still struggle to automatically craft abstract and complex rules. Integration of Logical Rules and LLMs Recently, the integration of inferential rules with neural networks, particularly LLMs, has gained significant attention. This approach combines the logical interpretability of symbolic reasoning and adaptive power of neural computing, improving LLMs’ logical reasoning ability. Wang et al. (2021); Olausson et al. (2023) transform textual statements into logical expressions and conduct symbolic reasoning following logical rules. Weir and Van Durme (2022) train neural models using a set of inferential rules for dynamic application. This direction broadens LLMs’ ability with flexible rule generation and application for complex reasoning.

heading: 6 Conclusion
text: This paper examines the proficiency of GPT-series LLMs in capturing logical inferential rules and
probes their challenging reasoning space. We introduce a logic scaffolding inferential rule generation (LOIRE) framework to create an inferential rule base ULogic, including nearly 8,000 primitive rules and over 6,000 compositional rules across five domains. Our evaluations using a subset of ULogic show that even advanced models like GPT4 struggle with compositional and structural complex rules and exhibit certain biases. Furthermore, we distill ULogic into a smaller inference engine that performs well in generating inferential rules and benefit downstream reasoning tasks. Our work points out where LLMs need to improve in logical reasoning and offers a pathway to enhance their reasoning capabilities.

heading: Limitations
text: Limitation on inferential rule coverage. Commonsense inferential rules may exist in diverse formats and span various domains. Our work mainly focuses on rules formatted as if-then statements, covering five domains: object affordance, accessibility, interaction, location and person’s need. In future work, we will expand our scope to include inferential rules of other formats and explore additional domains for broader coverage.
Limitation on probing open-source models. Our work does not probe and analyze open-source models. While GPT-4 and GPT-3.5-turbo are considered as the most advanced models, open-source counterparts may exhibit different behaviors or patterns in understanding inferential rules with varying complexities. These aspects will be the subject of future exploration.
Risk of environmental impact A significant risk associated with our framework and analysis is the potential increase in environmental burdens due to the extensive use of OpenAI’s APIs for LLMs. This impact can be mitigated by replacing GPT-4 with future smaller-scale open-source models that are more efficient with less environmental impact.
Potential error in rule generation. Generating inferential rules with specific requirements poses a significant challenge. As the majority of our framework’s pipeline are powered by GPT-4, it may inevitably generate inferential rules with logical inaccuracies even incorporating human verification. This might result in less accurate probing of LLMs.

heading: Ethical Consideration
text: All rules we collected through LLMs are released publicly for usage and its probing subset for proficiency analysis have been subjected to a thorough review by the authors. The code of our generation pipeline and probing experiments will also be publicly released. This setting guarantees transparency and reproducibility in our experiments, allowing other researchers to evaluate and expand upon our work. Our logic scaffolding framework is strictly limited to be used for rule generation that follow the ethical guidelines of the community. The authors emphatically denounce the use of our framework for generating inaccurate or harmful rules.

heading: Acknowledgments
text: Siyuan Wang’s research is supported by National Natural Science Foundation of China (No. 62176058) and National Key R&D Program of China (2023YFF1204800). The project’s computational resources are supported by CFFF platform of Fudan University. Xiang Ren’s research is supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via the HIATUS Program contract #2022-22072200006, the Defense Advanced Research Projects Agency with award HR00112220046, and NSF IIS 2048211.

heading: A Primitive Rule Generation Pipeline

heading: A.1 Abstract Objects and Common Properties
text: Table 5 list 32 most common abstract objects and 18 common properties for primitive rule generation.

heading: A.2 Rule Domains
text: Table 6 illustrates the detailed explanations, example predicates and rules across five domains.

heading: A.3 Prompts for Premise Generation
text: For premise generation in each domain, we design an instruction followed by two demonstrations to iteratively prompt GPT-4, and the underlined sentence is the rule description which varies according to the specific domain, as shown in Table 7.

heading: A.4 Grammatical Validity for Rule Filtering
text: As Figure 10, we check whether the variables in premises form a connected graph from node “X” to node “Y” to filter grammatically invalid rules.

heading: A.5 Prompts for Rule Filtering
text: Table 8 is an example prompt for rule filtering in object affordance domain.

heading: A.6 Human Verification Templates and Rates
text: Before human verification, we first craft a qualification task to select AMT annotators from all English-speaking countries (US, UK, New Zealand, Australia, Canada). The prospective workers are presented with three representative test cases and need to predict whether the premise and conclusion are clearly readable, and if the premise logically entails the conclusion. Only those workers correctly passing all the test cases are recruited. We compensated workers with different rates based on the
complexity of the annotated rules, with prices ranging from 0.08 to 0.16 dollars (i.e., 0.08, 0.10, 0.12, 0.14, 0.16) for rules with complexity levels from 1 to 5. On average, this amounts to 0.12 dollars per annotation. Each annotation takes approximately 0.5 minutes to complete, aiming to match a rate of 15 dollars per hour based on their working time.
The detailed template for human verification is shown as Figure 11. This template is also used for getting human performance in rule probing analysis, wherein a separate cohort of workers is quali-
fied for manual rule probing. Besides, the overall rates of rule acceptance in different domains during human verification are listed Table 9.

heading: A.7 Ablation on Rule Filtering
text: We calculated the rule yield rates for both heuristic filtering and self-critic, which are respectively 77.59% and 80.39%. Together, they totally yield 62.38% of candidate generated rules through the whole rule filtering process. We sample 100 rules before the self-critic and obtain 79 rules after the self-critic, and conduct human verification over them. Before the self-critic, 42 out of 100 rules (42%) are valid. After the self-critic, 40 out of 79 rules (50.63%) are valid. Among the 21 filtered rules, our self-critic can effectively identify 19 invalid rules, thereby enhancing the quality of retained rules.

heading: B Rule Probing

heading: B.1 Rule Probing Templates
text: Table 10 lists five different templates for unbiased rule probing.

heading: B.2 Dual-side Rule Probing Setting
text: Table 11 illustrate a concrete example of dual-side rule probing.

heading: B.3 Rule Depths Probing
text: The analysis of GPT-series LLMs and human on different compostional depths is presented as Figure 12. “Depth=0” represents primitive rules and “Depth=1,2,3” denote compositional rules involving 1 to 3 backward chaining steps.
B.4 Illustrations of Rule Structures Figure 13 displays several examples showcasing both symbolic and verbalized rules across different structure types.

heading: B.5 Different CoT Prompts
text: Table 12 lists different prompts of three CoT strategies for rule probing.

heading: B.6 Necessary Bias
text: As mentioned in Section 3.2, We investigate why GPT-4 significantly underperforms GPT-3.5-Turbo on transitive rules. Transitive rules typically describe a straightforward logical chain from variable X to Y, where GPT-3.5-Turbo and GPT-3.5-Instruct manage with greater ease. In contrast, we find that GPT-4 exhibits a “necessary bias” that tend to consider all necessary conditions reaching a conclusion, which avoids drawing a definite judgement. This conservative response is more pronounced in transitive rules, where GPT-4 more frequently responds with hesitations like "it does not necessarily mean" in its explanations. We present a probing example of a transitive rule by GPT-4 in Table 13, including its generated prediction, explanation and the corresponding label. We hypothesize that this conservative style may come from LLMs’ preference alignment during Reinforcement Learning with Human Feedback (Ouyang et al., 2022).
C Inference Engine
C.1 Illustration of Instruction Tuning
Figure 14 illustrate the pipeline of instruction tuning for rule distillation as an inference engine. Our inference engine is trained for three tasks: conclusion generation, premise completion and premise generation. The conclusion generation focuses on creating a conclusion from a provided premise. For premise completion, given a conclusion and its partial premise, the inference engine must complete the remaining premise part to support the conclusion. In premise generation, the engine is tasked with creating premises of varying complexity based on a given conclusion, specifically generating premises with one, two or even more facts. We also provide an inference engine demo for flexible rule generation as shown in Figure 15.
C.2 Implementation Details
We fine-tune Mistral-7b with our constructed instruction dataset with Quantization LoRA (QLoRA) method (Hu et al., 2021; Dettmers et al., 2023) as our inference engine. We set the learning rate to 7× 10−5, batch size to 8, gradient accumulation step to 16, and train the model 2 epochs. We apply QLoRA to all the linear layers of the model,

heading: Example
text: including q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj, and lm_head. The α and r of the QLoRA method are both set to 16.

heading: Inference Engine

heading: C.3 Prompting ChatGPT and GPT-4 for Three Tasks
text: As Step-2 of Sec. 2.2, we utilize two-shot prompts to instruct ChatGPT and GPT-4 in simultaneously generating symbolic and verbalized responses for three tasks, as shown in Table 14, 15, 16.

heading: C.4 Evaluation Metrics
text: We detailed describe the metrics for evaluating our inference engine against ChatGPT and GPT-4 for the premise generation task. • Accuracy: The premise is logically correct to
infer the conclusion and follow the instruction regarding the specific number of facts.
• Diversity: The degree of variation among the three generated rules.
• Complexity: Assessed only for premise generation with more than 2 facts, measuring the fact number and the semantic difficulty.
• Abstractness: The variable types in premises
are abstract to generalize to diverse instances. For example, the variable types “ Region” and “Event” are abstrct while “New York” and “The FIFA World Cup” are specific entities with low abstractness.

heading: C.5 LLM Evaluation Prompts
text: We prompt GPT-4 as the evaluator for rating the accuracy of conclusion generation and premise completion tasks, and the accuracy, diversity, complexity and abstractness of the premise generation task. We adopt one-shot prompts which are shown as Table 17 and Table 18 (with demonstrations omitted).

heading: C.6 Human Evaluation Templates
text: For the human evaluation of premise generation accuracy, we qualify a new cohort of AMT annotators to pairwise compare two sets of generated premises in terms of logical consistency with the provided conclusion, alignment with common sense and the inclusion of an accurate number of facts. The detailed template for human evaluation is shown as Figure 16.

heading: C.7 Downstream Reasoning Datasets
text: StrategyQA and SOCIAL IQA consist of crowdsourced questions involving reasoning of implicit logic. LINK comprises GPT-4 generated statements instantiated from abstract rules, including two subsets: head distribution statements and longtail knowledge statements. PIQA examines opera-
tional commonsense for achieving physical goals and CSQA2.0 features adversarial commonsense examples designed to mislead AI systems.

heading: Future Work
text: In future work, we will expand our scope to include inferential rules of other formats and explore additional domains for broader coverage. Limitation on probing open-source models. Our work does not probe and analyze open-source models. While GPT-4 and GPT-3. 5-turbo are considered as the most advanced models, open-source counterparts may exhibit different behaviors or patterns in understanding inferential rules with varying complexities. These aspects will be the subject of future exploration. Risk of environmental impact A significant risk associated with our framework and analysis is the potential increase in environmental burdens due to the extensive use of OpenAI’s APIs for LLMs. This impact can be mitigated by replacing GPT-4 with future smaller-scale open-source models that are more efficient with less environmental impact. Potential error in rule generation. Generating inferential rules with specific requirements poses a significant challenge. As the majority of our framework’s pipeline are powered by GPT-4, it may inevitably generate inferential rules with logical inaccuracies even incorporating human verification. This might result in less accurate probing of LLMs