LogicTree: Improving Complex Reasoning of LLMs via Instantiated Multi-step Synthetic Logical Data
Authors: Zehao Wang
Lin Yang
Jie Wang
Kehan Wang
Hanzhu Chen
Bin Wang
Jianye HAO
Defu Lian
Bin Li
Enhong Chen
Venue: NeurIPS 2025 spotlight ()
Keywords: large language model
logical reasoning
data synthesis
Primary area: deep_learning

Abstract: Despite their remarkable performance on various tasks, Large Language Models (LLMs) still struggle with logical reasoning, particularly in complex and multi-step reasoning processes. 
Among various efforts to enhance LLMs' reasoning capabilities, synthesizing large-scale, high-quality logical reasoning datasets has emerged as a promising direction. 
However, existing methods often rely on predefined templates for logical reasoning data generation, limiting their adaptability to real-world scenarios. 
To address the limitation, we propose **LogicTree**, a novel framework for efficiently synthesizing multi-step logical reasoning dataset that excels in both complexity and instantiation.
By iteratively searching for applicable logic rules based on structural pattern matching to perform backward deduction, **LogicTree** constructs multi-step logic trees that capture complex reasoning patterns. 
Furthermore, we employ a two-stage LLM-based approach to instantiate various real-world scenarios for each logic tree, generating consistent real-world reasoning processes that carry contextual significance.   This helps LLMs develop generalizable logical reasoning abilities across diverse scenarios rather than merely memorizing templates.
Experiments on multiple benchmarks demonstrate that our approach achieves an average improvement of 9.4\% in accuracy on complex logical reasoning tasks.

LogicTree: Improving Complex Reasoning of LLMs via Instantiated Multi-step Synthetic Logical Data
Zehao Wang'; Lin Yang", Jie Wang Kehan Wang', Hanzhu Chen',
Bin Wang
Jianye Hao Defu Lian', Bin Li', Enhong Chen
MoE Key Laboratory of Brain Science and Education, Psychological and Cognition, University of Science and Technology of China
Noah's Ark Lab, Huawei Technologies, 3 Tianjin University
Abstract
Despite their remarkable performance on various tasks, Large Language Models (LLMs) still struggle with logical reasoning, particularly in complex and multi- step reasoning processes. Among various efforts to enhance LLMs reasoning capabilities, synthesizing large-scale, high-quality logical reasoning datasets has emerged as promising direction. However, existing methods often rely on pre- defined templates for logical reasoning data generation, limiting their adaptability to real-world scenarios. To address the limitation, we propose LogicTree, a novel framework for efficiently synthesizing multi-step logical reasoning dataset that excels in both complexity and instantiation. By iteratively searching for applicable logic rules based structural pattern matching perform backward deduction, LogicTreee constructs multi-step logic trees that capture complex reasoning pat- terns. Furthermore, we employ a two-stage LLM-based approach to instantiate various real-world scenarios for each logic tree, generating consistent real-world reasoning processes that carry contextual significance. This helps LLMs develop generalizable logical reasoning abilities across diverse scenarios rather than merely memorizing templates. Experiments on multiple benchmarks demonstrate that our approach achieves an average improvement of 9.4% in accuracy for LLMs on complex logical reasoning tasks.
Introduction
Logical reasoning is crucial capability for large language models (LLMs) [5 24 14. providing substantial support for complex tasks such as coding, mathematics, and other higher-order cogni- 45). Recent advancements in the reasoning capabilities of LLMs have been impressive 46 with one of the key progress attributed to the availability of high-quality reasoning data 18
tive abilities [30] 28
6 Despite these strides, LLMs still encounter challenges in complex multi-step 31 42), underscoring a significant scarcity high-quality multi-step logical reasoning datasets.
9 50
logical reasoning tasks
Early logical reasoning datasets are primarily constructed through manual data collection and annota- tion [19 55 [3]. While these datasets are often diverse and intricate, their creation is labor-intensive and consuming for further training. In recent years, several studies have explored data synthesis approaches to generate logical reasoning data [37]. Some studies have employed set of atomic logic rules to evaluate and enhance the formal reasoning abilities of LLMs 41 33]. Similarly, other research has synthetically created multi-step logical reasoning datasets by leveraging natural language
templates 30 29 7 2. Furthermore, there is an increasing interest in generating reasoning processes
"Tii word done when Zehao Wang interned Huawei. Email: zh-wang mai..ustc.edu.co Conreponndigg author. jiewann @ ustc.edu.cn
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

for existing complex logical reasoning questions [21 [3]] However, existing data synthesis methods typically exhibit the following limitations: (1) insufficient complexity, as they typically generate simplistic reasoning patterns with limited types of reasoning rules and shallow reasoning steps, and (2) inadequate real-world scenario instantiation, often resulting from the combination of entities that lack contextual or conceptual relevance, which has been shown to weaken the robustness and generalization capabilities of LLMs in reasoning tasks 47]. Real-world scenario instantiation refers to the process of grounding synthetic reasoning data in concrete and context-rich world scenarios that incorporate semantically relevant entities or events, capable of reflecting the complexity and diversity of real-world reasoning tasks.
To address these challenges, we propose LogicTree, novel framework for efficiently synthesizing multi-step logical reasoning dataset, offering significant ad advantages in terms of complexity and instantiation. Our method leverages first-order logic rules to generate complex multi-step logical reasoning trees, which are then instantiated using I.LMs to produce natural language reasoning questions with real-world scenarios and reasoning processes that carry contextual significance. The logic trees generated by Logic exhibit complex reasoning patterns (as presented in Tables [0] and [I] facilitating the improvement of advanced reasoning abilities in LLMs for tackling complex tasks (Sec 4.3) Moreover, contextually diverse instantiated reasoning data help LLMs develop generalizable logical reasoning skills, rather than simply memorizing implicit relationships between facts 24 51 40).
Figure L presents an overview of Logic illustrating the data synthesis process in detail. Firstly, we construct symbolic logical reasoning trees via backward deduction procedure, where leaf nodes are iteratively expanded by applying formal logic rules identified through structural pattern matching of formulas. In this way, we generate multi-step logical reasoning trees that incorporate diverse and complex reasoning patterns, encompassing both propositional logic and first-order logic. Secondly, we introduce two-stage LLM-based approach to instantiate logical reasoning trees: (1) assigning contextually relevant entities or events to the logical symbols in the leaf nodes to construct realistic reasoning scenarios; and (2) sequentially translating all intermediate nodes in the logical tree to generate step-by-step natural language reasoning processes. By controlling the thematic domains of the content generated by LLMs, instantiate each logical reasoning tree with multiple diverse scenarios, thereby enhancing the overall diversity and contextual richness of the dataset (see Figures 3) Finally, apply post-processing to the synthesized data to verify the logical consistency of the instantiated content and construct logical reasoning instances, each consisting of a set of premises, conclusion-ani -answer pair, and the corresponding reasoning process. Experiments conducted across wide range of LLMS demonstrate that the instantiated complex logical reasoning data synthesized by Logic effectively enhances the models' reasoning capabilities. Furthermore, our analytical experiments (Table 3 indicate that diverse instantiation scenarios are beneficial for improving the generalization of reasoning abilities.
The main contributions of this work include:
We introduce LogicTree, novel framework for synthesizing multi-step logical reasoning datasets by generating complex logic trees with first-order logic rules and instantiating them into realistic reasoning scenarios, enabling LLMs to develop advanced and generalizable logical reasoning abilities (Sec 3).
We present a two-stage LLM-based instantiation technique that injects realistic statements into the symbolic reasoning trees, generating coherent and contextually grounded natural language reasoning processes (Sec
Experimental results demonstrate that the synthesized dataset significantly enhance the logical reasoning performance of LLMS across several challenging benchmarks, leading to an average accuracy improvement of up to 9.4% (Sec 4p.
2 Preliminary
Logical reasoning is a cognitive process that involves deriving valid conclusions from given premises based on formal logic rules [5
27 5. It is a comerstone of fields like artificial intelligence, mathematies, and philosophy, enabling systematic decision-making and problem-solving! 24 34
The main formal logic systems include propositional logic and first-order logic(FOL). Propositional

---

Logical Tree Generotion
Instantiation via LLM
4. Instantiation
Leaf Node List
Diversity
Logical Rules
Yx (-M))R))
Syllogiem
vz (=(-R(s) p- 7 T- Q PV (R(0))
topkss (num-3)
randomly sconarios
Public Health
Syllogiem
Stop @
Instandation
Instantiated Realistic Reasoning Scenarios
Modus Pontrs
powernet riissed natirmacee vazzine gempaggn"
(R()))
'Raa"" "lleson achareed eccusetion
T:: "Vacdhation rates 201066 regions incresse digificanly.
Inode Sequence based on depth
Search for Applicable Rules
smpt: T). 0)
Ponons
Doduction
Stopd: (P.(-R(a)) P.
Stop
root node
inode
leaf node
Natural Language Reasoning Processes
Synthetic Example Post- processing
rule -R/x),,
achioved
Premises:
coverage Therefore, City
"rule1": region achieved vaccination ovverage."
vacciration coverage [*][a]].
government inttiated nationvice vaccination campaigh then vaciination rates acrosd regions increased s signiicantty"" "rule3": "rule3":.., ..., "rule"":..
Step 2: Using rule (P + 1) and rule (T + QJ, we deduce that the iniiiated nationwide veccination (C).
(P), then
effectively
conclusions:
Step 3: From rule IP Ala) and step1 ["aa]]], we deduce that the government initiated nationwide veccination campaign (P). deduce that spread
"The spread the disease effectioell contained." True
"Country is making efforts develop
vehicle industry.
rerrom Q) was effectively contained
City Z
vaccination coverage."
False
Figure 1: An overview of LogecTree, which comprises three key modules: (1) Logical reasoning
tree generation via iterative backward deduction based on structural pattern matching; (2) Reasoning scenario instantiation using two-stage LLM-based approach; (3) Synthetic reasoning example post-processing.
logic focuses on propositions or statements that are either true or false, employing logical operators such A (and), V (or), (not), and -> (implies) to connect these statements. In contrast, FOL extends propositional logic by handling more complex statements involving variables, quantifiers (e.g., V for "for all" and 3 for "there exists"), and predicates.
In formal logic system, a logical expression composed of the aforementioned symbols is referred to formula (e.g.. A - B, vzP(z)). If an expression contains no logical operators, it is called an atomic formula. By representing premises or statements as formulas, logical rules can be systematically applied to derive new conclusions. For instance, Universal Instantiation and Hypothetical Syllogism are two well-known inference rules:
UI: vx(p(z)) + p(a)
HS:
(2)
Here, + denotes "derives", meaning that the formulas preceding it serve as premises, while the formula following represents the conclusion. The Hypothetical Syllogism (HS) rule states that if "p implies q" and "g implies r", then we can conclude that "p implies r". For additional logical rules and further details, please refer Appendix
3 Method
In this section, introduce the design and implementation of LogicTTre,, novel framework for synthesizing instantiated complex logical reasoning datasets. Figure overviews the workflow of our approach, which consists of three main modules: logical reasoning tree generation, reasoning scenario instantiation and synthetic reasoning example post-processing. First, based on structural pattern matching, we construct symbolic logical reasoning trees through a backward deduction pSecess(Sec., 3.1) Next, we leverage LLMs to instantiate the logical reasoning trees with scenarios (Sec 3.2). Finally, we verify the logical consistency of the instantiated content and generate synthetic logical reasoning instances(Sec 3.3).
3

---

3.1 Logical Reasoning Tree Generation
Logical Reasoning Tree Each node in logical reasoning tree (logic tree) corresponds to logical expression (formula) that can be derived from its child nodes based logic rules. The role of each node varies depending on its position within the tree: leaf nodes represent premises, the rool node denotes the final conclusion, and intermediate nodes (inodes) encapsulate individual reasoning steps.
In order to construct logical reasoning tree,we we designed a backward deduction method based on structural pattern matching of formulas, which starts from given formula and searches for applicable logic rules to derive the premises that support it. When determining whether logic rule can be applied formula, this method does not require the formula to be identical the conclusion of the logic rule; instead, it only compares whether the structural patterns of their are matched. In this way. we generate multi-step logical reasoning trees that incorporate diverse and complex reasoning patterns, encompassing both propositional logic and first-order logic. We provide additional detailed information the Appendix C.2 including the types of logical rules involved and the complexity of the logical trees (e.g., depth, breadth, and the number of rules).
Algorithm Logical Reasoning Tree Generation Process
1: Input: Root conclusion formula Q, number of iterations N, rules list R {ri, 72 ................................ Fmn
2: T init_ Init_logictree(
3: k ==1
4: while k < N do
5: L random_seleen select leaf nT)e(T)
6:
A 0 #uppliable rules list
7:
for R do
8:
if is ts_appliabll_riee ciciee_rule(_a wwle(C.AST, r.comclusion.AST) then At Au {r} end if end for
9:
10:
12:
appliable culleee_rllle---ee handom_choicel (erruarddddductionn( k == ++1
L.childs
14:
15:
end while
16: Output: Final logical reasoning tree T
Structural Pattern Matching of Formulas Logical formulas can be represented as Abstract Syniax Trees (ASTs). this AST representation, root node typically corresponds to the logical operator with the highest precedence. The operands of this operator are then represented by the subtrees branching from the root. This decomposition process is applied recursively to the subtrees, continuing until the tree's s nodes represent only atomic formulas, which form the leaves of the tree. The structure of a formula considered to match the structural pattern of another if their AST's respective operator skeletons are isomorphic. This means their ASTs have the same upper-level structure. For example, consider the formulas (A v B) - (-C) and F - G. Although the two formulas not identical, they have the same outermost logical operator, namely are the conclusions of Rule (2), we can apply this rule to (A V B) + (-C) for backward deduction, resulting in the premises (A V B) - G and G (-C). Therefore, if the conclusion formula of rule matches the structural pattern of the target formula, the rule can be applied to the target formula. Refer Appendix C.T and Figure for more details.
Since F + G
Multi-step Backward Deduction Generation As shown in Algorithm I our logical reasoning tree generation follows backward deduction process, iteratively searching for applicable formal logical rules based on structural pattern matching to decompose existing formula nodes into new ones. Initially, the logical tree consists of single root node, which is randomly generated formula. Then, the process iterates as follows until the desired number of iterations is reached: Randomly select leaf node the tree, apply relevant rules to perform backward deduction, and derive the necessary premises, which are added as child nodes to the selected formula. The number of iterations and the search strategy can control the structure of the final logical reasoning tree, specifically its depth and width. Examples of logical reasoning trees can be found in the Appendix E

---

3.2 Reasoning Scenario Instantiation via LLMs
Previous works m00 29 3 have employed template based approach to translate the symbolic formulas into natural language statements. However, these methods fail to capture the contextual semantics of logical relationships between formulas, often generating statements with limited real-world applicability.
Thus, we aim to instantiate logical reasoning trees with realistic reasoning scenarios, which involves translating symbolic legical expressions into concrete statements that carry contextual significance. Given the advanced text generation capabilities of modern LLMs (c.g.. GPT-4 M., an intuitive approach is to leverage LLMs for instantiation. However, due to the inherent limitations of LLMs in handling complex logical reasoning, directly prompting them to generate realistic logical reasoning problems from symbolic logic trees may introduce errors. To mitigate this issue, we design the following two-state prompting strategy to guide LLMs in instantiating the logical reasoning tree. The details of prompts provided in Appendix D
(1) Logical tree instantiation: We extract all the formulas that are leaf nodes in the logical reasoning tree and input them into LLMs to instantiate a real-world reasoning scenario. During this process, LLMs assign real-world statements atomic formulas while ensuring consistency with their predefined logical relationships. Once all leaf nodes are translated into natural language, a coherent and contextually grounded logical reasoning scenario formed.
(2) Reasoning process translation: We obtain a complete, step-by-step symbolic reasoning chain by sorting the internal nodes of the logical tree descending order of depth. At each step, an inode is derived from its child nodes according to the corresponding logical rules. We then leverage LLMS to translate the symbolic reasoning steps into a natural language reasoning process within the previously instantiated scenario.
In this process, the LLM is guided by correct and rigorous symbolic reasoning skeleton to generate a step-by-step, instantiated natural language reasoning process. This approach signifi- cantly reduces generation errors, as the LLM is not required to perform the reasoning autonomously; it only needs to understand the logical relationships and perform symbolic translatio, which is well within the capabilities of LLMs 32 36). Furthermore, by controlling the themes of the reasoning scenarios used by the LLM, we can ensure diversity in the synthesized data (Figure 4. This approach enables the generation of coherent, step-by-step logical reasoning processes in natural language that incorporate a wide range of realistic scenarios.
3.3 Synthetic Reasoning Example Post-processing
We conduct systematic verification of the logical consistency of the instantiated content to filter out erroneous data. Specifically, this involves prompting the L.MM to output the instantiated natural language statements paired with their corresponding logical expressions, typically in the format "statement [expression]". We then perform an exact string comparison between the logical expression provided by the LLM and the original source logical expression to identify any inconsistencies. The entire natural language reasoning process is validated once all statements are confirmed to be logically consistent with their symbolic expressions, as the overall correctness is inherently guaranteed by the logic tree. This rigorous verification process enables us to filter out the data identified as erroncous, thereby significantly improving the overall data quality.
Then, we utilize the verified instantiated content to construct logical reasoning data instances. A logical reasoning instance is composed of a set of premises, a conclusion-ansee answer pair, and the corresponding reasoning process. In particular, the natural language statements of the leaf nodes are combined to form the premises. Then we can construct conclusions with different types of answers ('True", "False" or 'Uncertain") based on the logical tree. For answer label "True". we use the statement of root node as the conclusion. For answer label "False", we use the negated statement of the root node as the conclusion. For answer label "Uncertain"", we introduce some irrelevant statements as distractors. Examples of synthetic logical reasoning data can be found in Appendix
5

---

Experiments
To evaluate the effectiveness our LogicTree, we design a suite of experiments that not only demonstrate its significant impact on enhancing L.L.s' reasoning abilities, but also provide in-depth analytical insights. Specifically, we divide the experiments into three parts:
To comprehensively assess the enhancements of our approach in improving the logical reasoning abilities of LLMs, we conduct evaluations across multiple benchmarks and compare it against several other synthetic logical datasets (Sec
To further evaluate the model's multi-step logical reasoning ability, assess Logic Tree Tree and baselines on Multi-LogiEval, which categorizes questions by reasoning steps from one five (Sec 4.3
To analyze the key factors contributing to LogicTree'ss effectiveness, we conduct ablation studies on three core components: (1) instantiating real-world reasoning scenarios, (2) incorporating natural language reasoning processes under instantiated scenarios, and (3) ensuring diversity in instantiated scenarios (Sec 4.5).
To further evaluate the generalization enhancement of model reasoning capabilities by LogicTree, conduct additional experiments across diverse range of task types, covering domains such as logic, math, code, NLI, and others (Sec|4.4).
To assess the generalizability of our synthetic data, we extended our evaluation to include models from different families and scales, specifically Qwen2.5-1.5B/31 and the Deepseek- RI R1-Distill series (Appendix B.3
4.1 Experimental Setup
We briefly explain the experimental settings. Refer to Appendix B for more details.
Syntheised Training Dataset. We generated 5,000 symbolic logic trees with depths from 2 to 15, and instantiated each into semantically diverse scenarios, yielding 15,000 reasoning problems. After applying automatic filtering process that discarded 8.73% of noisy invalid samples, the final dataset for LLM training contains 13.8k high-quality, multi-step reasoning instances.
Evaluation Benchmarks. To evaluate the effectiveness of our proposed LogicTree, we consider diverse set reasoning tasks and datasets that require rigorous logical reasoning: (a) LogieBench [33 novel task designed to comprehensively evaluate the model's performance on each inference rule. (b) A..iQ
collection of challenging real-world logical reasoning problems from civil service entrance exams. (c) Three BIG-Bench Hard (BBH) 38 tasks of varying difficulty levels: logical deduction with three, five, and seven objects. (d) FOLIO [3]: An expert-written, logically complex dataset for first-order logic reasoning. (e) Two AGIEval [55] tasks focused on logical reasoning: LAST-AR and LAST-LR. (f) Multi-LogiEEval [ 35): comprehensive dataset incorporating varying reasoning depths for logical complexity.
Baselines. To demonstrate the superiority of our method, we evaluated the following baselines. (i) Vanilla: standalone LLMs without additional training. Vanilla LLMs represent the original capabilities of LLMs.. (ii) PARARULE(]]: it generates deductive processes by repeatedly applying deduction rules to a given set of facts and transforms these processes into sentences based on natural language templates. (iii) LogicAsker(4I]: a framework to evaluate the ability of LLMs to handle individual atomic logical reasoning rules and generate targeted samples for improvement. (iv) FLD, [29]: it introduces systematic design principles for logical data synthesis and randomly combines various rules to construct multi-step reasoning datasets.
Models and Training Settings. To rigorously validate the effectiveness of LogicTree, we conduct extensive experiments using diverse set of prominent open-source models spanning various families and scales. We utilize models from the 1lama-3.1, Mistral-v0.3. Qwen2.5, and Deepseek-RI ---Disilll families, with parameter scales ranging from 1.5B to 70B. We employ two distinct fine-tuning strategies: full fine-tuning applied to smaller-scale models (under 8B), while the larger 70B model utilizes LoRA fine-tuning.

---

Table 1: Main results five types of reasoning tasks. A means the margin between Vanilla and training by LogicTree. "Avg" means the average accuracy across five benchmarks for each methods. We bold the best results for each LLM backbone and underline the second-best results.
AGIEval
Model
LogicBench
LogiQA2.0
FOLIO
BBH--Lggi
Avg.
LR
AR
Llama-3.1-8B
vanilla
80.010.4
42.410.2
50.8_0.4
39.310.3
48.410.3
20.710.1
46.9
+ PARARULE
84.4
53.3+0.2
50.3_0.s
42 42.4go.1
52.9
21.7+0.2
50.8
LogicAsker
85.0
54.7
54.6...2
43.7
51.210.2
23. 10.1
52.0
Fl.Dx2
83.620.2
53.3+0.2
54.520.3
39.520.2
51.8+0.3
21.320.1
50.7
+ LogicTree
90.6g0.2
53.5
57 57.620.2
53.220.2
56.1g0.1
26.5_0.2
56.3
+10.6
+11.1
+13.9
+7.7
+5.8
+9.4
(Relative Gain)
(+13.3%)
(+26.2%)
(+13.4%)
(+35.4%)
(+15.9%) (+28.0%)
(+20.0%)
Qwen-2.5-7B
vanilla
83.1
63.810.4
54.320.4
72.820.3
65.320.3
23.910.2
60.5
PARARULE
82.9
65.1 +0.1
55.3=p.2
72.8+0.4
68.3
24.5+0.1
+ LogicAsker
84.3
64.9_o. 0..1
58.7
72.810.2
66.710.2
24.5
62.0
+ FL
83.4
65.1
59.610.3
73.610.2
67.310.1
26.1 10.1
62.5
LogicTree
89.2go.1
64.9+a.1
63.8 63.8g0.1
74.3
69.3
28.7=0.2
65.0
A
+6.1
+1.1
+9.5
+1.5
+4.0
+4.8
+4.5
(Relative Gain)
(+7.3%)
(+1.7%)
(+17.5%)
(+2.1%)
(+6.1%)
(+20.1%)
(+7.4%)
Mistral-7B-v0.3
vanilla
77.5
49.7
51.220.2
45.5
49.920.3
20.4g0.1
49.0
PARARULE
78.8
51.2+0.3
50.3
47.220.2
50.410.2
22.610.2
50.1
LogicAsker
78.8
50. 50.6+a.2
70.7
46.8+0.1
50.2+0.2
23.1
51.0
FL.xx2
78.010.2
50.610.2
54.5_o.3
48.1
50.4
22.210.1
50.6
LLogicTree
81.9
52.3 0..1
52.510.2
48.7.0.2
51.4
25.7_0.2
52.1
A
+4.4
+2.6
+1.3
+3.2
+1.5
+5.3
+3.1
(Relative Gain)
(+5.7%)
(+5.2%)
(+2.5%)
(+7.0%
(+3.0%)
(+26.0%)
(+6.3%)
1ama-331-70B
vanilla
91.320.2
67.1
59.620.4
75.320.2
82.240.1
28.32o.1
67.3
PARARULE
92.5
68.2
61.120.3
75.6.0.1
82.4+0.2
26. 26.1
67.7
LogicAsker
13.3
69.91..2
63.4_p.2
75.210.1
82.810.2
30.8+o.1
69.2
FL.
93.7+
69.9 4-..1
63.12002
74.8
83.1
26.1
68.5
LogicTree
94.410.2
70.3_o.
65. 65.210.2
76.0.0.0
83.7
31.3_m
70.2
+3.1
+3.2
+5.6
+0.7
+1.5
+3.0
+2,9
(Relative Gain)
(+3.4%)
(+4.8%)
(+9.4%)
(+0.9%
(+1.8%)
(+10.6%)
(+4.3%)
4.2
Main Results
To validate the significant improvement in logical reasoning abilities enabled by our synthesized data, we evaluate LogicTree on multiple downstream logical reasoning tasks. As highlighted in Table 1, LogicTrre shows consistent and substantial performance improvements across all backbone models and most benchmarks against the baselines. demonstrating the effectiveness of instantiating complex logical reasoning trees. Specifically, LogicTree surpasses vanilla models on all benchmarks, achieving significant margins. On LogicBench and AGIEval-LR, improves LLama-3.1--8 by 10.6% and 7.7%, respectively. Even on AGIEval-AR, particularly challenging benchmark for LLMs, our approach still achieves 6% improvement. Meanwhile, our method achieves average improvements of 9.4%, 3.1 %, and 2.9% on Llama-3.1 1-B8B, Mistral-7B-v0.33 and 1lama-3.1-70% respectively, underscoring robustness across different model architectures and scales. Additional experimental results more models can be found in Table 5 and

---

Results of Libma 1-80 1-88-lnctrud en Nutth Logibunl
Results of Quese2 1-78--estnud
instnuct on Hult-Legiival
Results ie Mulli-Gaplhell
90
Acourscy
70
Accurccy
60
700
60
50
Viuti
50
40-
40
1.0"2
FLD"2
LogiTeo
che
JogicTus
61
d2
85
d1
02
4J
02
63
64
45
Reasoning Depth
Reasoning Depth
Reasoning Depth
Figure 2: Experiment results for different methods on Multi-LogiEval, which is specifically designed to evaluate the model's multi-step logical reasoning abilities. The horizontal axis represents the number of reasoning steps required to answer the question, ranging from dj to ds- The bold red line the top represents the number of reasoning steps increases.
method, demonstrating superior performance improvements, especially as
Table 2: Experiment results using Llama3. 1-8B across broader range of task types, covering
domains such as logic, math, code, NLI, and others.
Model
Proofwriter
Mathqa
GPQA
Humaneval
Commencenseqa
MNLI
Avg.
Llama-3.1-8B
vanilla
57.3
42.1
30.8
68.4
73.9
68.1
56.8
PARRRULE
64.3
42.8
31.9
67.2
73.7
70.5
58.4
LogicAsker
69.8
43.2
30.1
68.6
74.5
73.3
59.9
+FLDx2
74.1
43.8
31.1
72.3
74.1
75.3
61.8
+LogicTree
76.9
47.3
32.8
74.6
75.9
76.8
64.1
worth noting that LogicTree demonstrates superior performance all baselines, especially in multi-step complex reasoning tasks. While LogicAsker generates large volume of atomic instruction data that provides some improvement in basic logical reasoning, it remains inferior to approach. Furthermore, the datasets synthesized by these baselines yield only marginal gains more challenging reasoning tasks, such as BBH-Logic and AGIEval-AR. In contrast, our method achieves performance gains of up to 13.9%, further demonstrating that complex reasoning patterns and instantiated reasoning processes play a crucial role in enhancing the model's reasoning capabilities.
4.3 Results of Multi-step Reasoning
To further evaluate the model's multi-step logical reasoning ability, we assess LogicTree and baselines on Multi-LogiEval, which categorizes questions by reasoning steps from one five. As shown in Figure LogicTree consistently outperforms all baselines across all categories of test data, with particularly strong improvements at higher reasoning depths (d3, d4, d5). These results indicate that LogicTree effectively enhances LLMs' ability to tackle complex multi-step reasoning tasks. In contrast, other methods exhibit sharp decline in performance as reasoning depth increases, highlighting their limitations in scaling to more complex reasoning scenarios. Their struggle to maintain accuracy multi-step reasoning tasks suggests that training with simple reasoning data alone insufficient for significantly improving the model's ability to handle complex reasoning. LogicTree's superior performance multi-step reasoning tasks underscores the potential benefits of its approach improving reasoning abilities on multi-step tasks, making it a valuable tool for complex logical reasoning scenarios. Results related to the 70B model can be found in the Appendix B.4
4.4 Results on Generalization Experiments Across Multiple Domains
To verify the generalization of LogicTree in enhancing model reasoning capabilities, we conducted additional experiments using 8B model across broader range of task types, covering domains such logic, math, code, NI.I, and others. shown in the table, LogicTree enhances the model's reasoning abilities across multiple domains. The improvement is smaller in tasks requiring extensive

---

Table 3: Ablation results four types of reasoning tasks, Llama-3. the backbone models. "Avg" means the average accuracy across four benchmarks for each methods. We bold the best results for each benchmark.
AGIEval
Model
LogieBench
LogiQA2.0
BBH
Avg.
LR
AR
Llama-3.1-8B
vanilla
80.0
42.4
39.3
48.4
20.7
46.2
+ LogicTrree (rem()um=3)
90.6
53.5
53.2
56.1
26.5
55.9
w/o sns__scenario
86.3
53.5
46.7
54.5
22.6
52.7
w/o inst reasoning
88.3
52.9
50.1
53.9
24.4
53.9
w/o inst diversity
num=1
51.2
49.6
54.1
23.9
52.4
num=2
88.1
52.9
50.8
52.9
26.1
54.2
num=4
92.6
52.7
53.2
56.1
25.7
55.2
num=5
94.5
50.3
50.8
51.2
23.7
54.1
knowledge recall, such as commonsense and science, since LogicTrree primarily strengthens gen- eralization rather than knowledge retention, which is largely dependent on pretraining. However, in ceasoning-intensin domains like logic, math, and code, the model exhibits significant gains, demonstrating that LogicTree effectively improves reasoning capabilities through diverse instantiated reasoning processes.
4.5
Ablation Study
In this section, we conduct ablation experiments to assess the contributions of key design components
in LogicTree (1) instantiating real-world reasoning scenarios (Analysis 1). (2) incorporating
natural language reasoning processes under instantiated scenarios (Analysis 2), and (3) ensuring diversity in instantiated scenarios (Analysis 3). Specifically, "w/o inst_ scceario" refers to training on symbolic logical expressions without instantiating the logical reasoning tree into real-world scenarios; "wfo inst reasoning" denotes instantiating real-world scenarios while retaining fully symbolic representation of the reasoning process; and "inst diversity" means omitting control over the diversity of instantiated scenarios. In particular, evaluate performance by instantiating one and two reasoning scenarios and processes per logical tree (whereas Logic' Tree employs three). As shown in Table removing any of these components results in notable performance decline, highlighting their importance in enhancing logical reasoning capabilities.
Analysis 1: Instantiating reasoning scenarios enables the model to learn the ability to apply logical reasoning in specific tasks, rather than simply memorizing language templates or the potential relationships between facts. As shown in Table while symbolic rules enhance reasoning, the improvement without instantiation significantly less than with LogicTrre. This underscores instantiation's 's importance: it teaches applying logic in complex tasks, not just memorizing rules.
The reasoning processes in natural language based on instantiated scenarios can further enhance the model's reasoning performance in challenging tasks. As Table demonstrates, instantiating symbolic logical problems into real-world scenarios improved the model's reasoning and accuracy compared to purely symbolic questions, supporting Analysis I's findings. However, its performance remained inferior to LogicTree. This underscores that natural language reasoning based instantiated scenarios is crucial for boosting logical ability in complex tasks. On hard problems like AGIEval-AR, this instantiation method yielded only modest gains 1.8%, highlighting the growing value of natural language reasoning on instantiated scenarios for intricate challenges.
Analysis 2:
Analysis 3: Diverse reasoning scenarios able LLMs to develop more generalizable and trans- ferable logical reasoning abilities. We instantiate different numbers of reasoning scenarios for the same logical reasoning tree, ranging from to 5. When num=1, the model's performance on some

---

benchmarks is even worse than when no instantiation is used ("wo instantiation"), This suggests that using only a single reasoning scenario may cause the model overfit to the relationships between the facts within that scenario, rather than learning genuine reasoning abilities. However, as num increases, the model's performance continues to improve and converges around 3 or 4. Therefore, incorporating more diverse reasoning scenarios proves beneficial for enabling the model to acquire genuinely generalized logical reasoning skills.
5 Related Works
5.1 Synthetic Logic Corpus for Training
9 50, there has been growing focus on generating high-quality logical reasoning datasets to enhance the reasoning capabilities of LLMs [3]] Early works [7] [3 3 generated deductive processes by repeatedly applying deduction rules to given set of facts and transformed these processes into natural language sentences using fixed introduced systematic design principles for logical data synthesis and randomly combined various rules to construct multi-step reasoning datasets. (author?) [4]] proposed framework to evaluate the ability of L.LMs to handle individual atomic reasoning rules and generate targeted samples for improvement. LogicPro [16] further advanced this line of research by using LLMs to convert algorithmic problems into logical reasoning tasks. However, These methods still have limitations in the complexity of logical trees, producing rules or facts with limited real-world applicability and simplistic language. Our research further extends the generation of logical reasoning trees, encompassing various complex reasoning patterns first-order logic. Additionally, we use LLMs to instantiate these trees into realistic reasoning scenarios and generate reasoing process in natural language.
With the increasing adoption of synthetic datasets 18
templates. FL 30 29
5.2 Evaluation of Logic Reasoning
Accurately evaluating the reasoning abilities of LLMs is both essential and challenging. Many studies have focused on assessing this fundamental capability of LLMs [20). For instance, LogiQA A [9]
and AR-LAST(551 evaluated models' logical reasoning abilities in real-world scenarios by collecting
human examination questions. LogicBench [33] demonstrated that existing LLMs struggle to handle
complex logical contexts, even when involving only single reasoning pattern. (author?) [41) introduced LogicAsker, which utilizes a set of atomic reasoning skills to assess the ability of L.MMs to learn logical rules. FOLIO [33] and Multi-Logi/Evl[[35 further assessed the multi-step logical reasoning capabilities of LLMs. These studies [8, [4] highlight the significant challenges faced by LLMs logical reasoning tasks, underscoring the critical need for high-quality training datasets to improve their reasoning performance.
Conclusion
In this paper, we propose LogicTrree, novel framework for synthesizing instantiated, complex logical reasoning datasets. Our method leverages first-order logic rules to generate complex multi-step logical reasoning trees, which are then instantiated using LLMs produce natural language reasoning data with realistic scenarios. The resulting synthetic datasets feature intricate reasoning patterns, promoting the development of advanced reasoning abilities LLMs address complex tasks. Furthermore, by incorporating instantiated reasoning processes, our method enables LLMs to acquire generalizable reasoning skills, rather than simply memorizing implicit relationships between facts. Extensive experiments across multiple complex logical reasoning benchmarks demonstrate the effectiveness of LogicTree, with an average accuracy improvement of 9.4%. LogicTree consistently outperforms all baselines, particularly in multi-step complex reasoning tasks. Further analysis indicates that both instantiated reasoning processes and the diversity of instantiated scenarios contribute to enhancing the model's generalizable logical reasoning abilities.
10

---

Acknowledgments
The authors would like to thank all the anonymous reviewers for their insightful comments and valuable suggestions. This work was supported by the National Key R&D Program of China under contract 2022ZD01 19801, and the National Nature Science Foundations of China grants U23A20388, 62021001 and 624B1011.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igge Akkaya, Florencia Leoni Aleman, Dingo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint ariix:2333.08775 2023.
[2] Qiming Bao, Alex Yuxuan Peng, Tim Hartill, Neset Tan, Zhenyun Deng, Michael Witbrock, and Jiamou Liu. Multi-step deductive reasoning over natural language: An empirical study on -ittt--distritttion gencralisation. arXiv preprint arXiv:2007.1 14000, 2022.
[3] Qiming Bao, Alex Yuxuan Peng, Tim Hartill, Neset Tan, Zhenyun Deng. Michael Witbrock, and Jiamou Liu. Multi-step deductive reasoning over natural language: An empirical study on out-of-distribune generalisation. arXiv preprint arXiv:2207. 14000, 2022.
[4] Leonardo Bertolazzi, Albert Gatt, and Raffaella Bernardi. A systematic analysis of large language models as soft reasoners: The of syllogistic inferences. arXiv preprint arXiv:2406.1134/ 2024.
[5] Hugo Bronkhorsi, Gerrit Roorda, Cor J. M. Suhre, and Martin J. Goedhart. Logical reasoning in formal and everyday reasoning tasks. International Journal of Science and Mathematics Education, 18: 1673-164,4, 2020.
[6] Hanzhu Chen, Xu Shen, Jie Wang, Zehao Wang, Qitan Lx, Junjie He, Rong Wu, Feng Wu, and Jieping Ye. Knowledge graph finetuning enhances knowledge manipulation large language models. In The Thirteenth International Conference on Learning Representations, 2025.
[7] Peter Clark, Oyvind Tafjord, and Kyle Richardson. Transformers as soft reasoners over language. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 3882-3890, 2021.
[8] John Dougrez-Lewis, Mahmud Elahi Akhter, Yulan Hc, and Maria Liakata. Assessing the reasoning abilitics of chatgpt in the context of claim verification. arXiv preprint arXiv:2402.1077 2024.
[9] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AL- Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
[10] Xidong Feng. Ziyu Wan, Muning Wen, Stephen Mareus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero-lie tree-search can guide large language model decoding and training. arXiv preprint arXiv:2309.1711 2023.
[11] Zijie Geng, Jie Wang, Ziqi Liu, Feng Ju, Yiming Li, Xing Li, Mingxuan Yuan, Jianye Hao, Defu Lian, Enhong Chen, and Feng Wu. Accurate KV cache eviction via anchor direction projection for efficient LLM inference. In The Thitt Annual Conference on Neurul Information Processing Systems, 2025.
[12] Runquan Gui, Zhihai Wang, Jie Wang, Chi Ma, Huiling Zhen, Mingxuan Yuan, Jianye HAO, Defu Lian, Enhong Chen, and Feng Wu. Hypertree planning: Enhancing 1lm reasoning via hierarchical thinking. In Forty-second International Conference on Machine Learning, 2025.
[13] Simeng Han, Hailey Schoclkopf, Yilun Zhao, Zhenting Qi Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujic Qiao, Luke Benson, al. Folio: Natural language reasoning with first-order logic. arXiv preprint arXiv:2209.0084) 2022.
[14] Ruixin Hong. Hongming Zhang. Xinyu Pang. Dong Yu. and Changshui Zhang. closer look at self-verification abilities of large language models in logical reasoning. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 900-925, 2024.
[15] Jie Huang and Kevin Chen-Chuan Chuan Chang. Towards reasoning in large language models: A survey, arXiv preprint 2rX.:22212.10403, 2022.

---

[16] Jin Jiang, Yuchen Yan, Yang Liu. Yonggang Jin, Shuai Peng. Mengdi Zhang, Xunliang Cai, Yixin Cao. Liangeai Gao, and Zhi Tang. Logiepro: Improving complex logical reasoning via program- -aridde--uuddddded learning. arXiv preprint arXiv:2409. 12929, 2024.
[17] Xize Liang, Lin Yang. Jie Wang. Yiyang Lu, Runyu Wu, Hanzhu Chen, and Jianye Hao. Boosting multi-domain fine tuning of large language models through evolving interactions between samples. In Forty-second International Conference on Machine Learning. 2025.
[18] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412. 19437, 2024.
[19] Hanmeng Liu, Jian Liu, Leyang Cui, Zhiyang Teng, Nan Duan, Ming Zhou, and Yue Zhang. Logiqa 2.0-an improved dataset for logical reasoning natural language understanding. IEEE/ACM Transactions on Audio, Speech, and Language Processing. 2023.
[20] Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. Evaluating the logical reasoning ability of chatgpt and gpt-4, arXiv preprint arXiv:2304.03433 2023.
[21] Hanmeng Liu, Zhiyang Teng, Leyang Cui, Chaoli Zhang, Qiji Zhou, and Yue Zhang. Logicot: Logical theuugh instruction-tuniin arXiv preprint arXiv:2305. 12147, 2023.
[22] Haoyang Liu, Jie Wang. Wanbo Zhang, Zijie Geng, Yufci Kuang, Xijun Li, Bin Li, Yongdong Zhang, and Feng Wu. stuPist MILP instance generation via block structure decomposition. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[23] Tongxuan Liu, Wenjjang Xu, Weizhe Huang, Yuting Zeng, Jiaxing Wang, Xingyu Wang, Hailong Yang, and Jing Li. Logic-of-thoughh Injecting logic into contexts for full reasoning in large language models. arXiv preprint (4X.:::449.173333 2024.
[24] Man Luo, Shrinidhi Kumbhar, Mihir Parmar. Neeraj Varshney, Pratyay Banerjee, Somak Aditya, Chitta Baral, al. Towards logiglue: A brief survey and benchmark for analyzing logical reasoning capabilities of language models. arXiv preprint arXiv:2310.00830 2023.
[25] Qitan Lv, Jie Wang. Hanzhu Chen, Bin Li, Yongdong Zhang, and Feng Wu. Coarse-to-fine highlighting: Reducing knowledge ballucination in large language models. In International Conference on Machine Learning, pages 33594 336233. PMLR, 2024.
[26] Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison- Burchh. Faithful chain-of of-thought reasoning. In The 13th Internatonnal Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL 2023), 2023.
[27] Bill MacCartney and Christopher D Manning. Natural logic for textual inference. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 193-200, 2007.
[28] Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. Gsm-symbolie: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv:4410.05229, 2024.
[29] Terufumi Morishita, Gaku Morio, Atsuki Yamaguchi, and Yasuhiro Sogawa. Enhancing reasoning capabilities of Ums via principled synthetic logic corpus. In The Thirty-eighth Annual Conference on Newral Information Processing Systems.
[30] Terufumi Morishita, Gaku Morio, Atsuki Yamaguchi, and Yasuhiro Sogawa. Learning deductive reasoning from synthetic corpus based formal logic. In International Conference on Machine Learning, pages 25254-25274. PMLR, 2023.
[31] Kentaro Ozeki, Risako Ando, Takanobu Morishita, Hirohiko Abe, Koji Mineshima, and Mitsuhiro Okada. Exploring reasoning biases in large language models through syllogism: Insights from the neubaroco dataset. In Findings of the 62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024, pages 16063-16600 16077. Association for Computational Linguistics (ACL), 2024.
[32] Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. Logic-Im: Empowering large language models with symbolic solvers for faithful logical reasoning. arXiv preprint 125. 2023.
12

---

[33] Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, and Chitta Baral. Logicbench: Towards systematic evaluation of logical reasoning ability of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 13679-13707, 13707, 2024.
[34] Barbara H. Partee, Alice Meulen, and Robert E. Wall. Mathematical methods in linguistics. 1990.
[35] Nisarg Patel, Mohith Kulkamni, Mihir Parmar, Aashna Budhiraja, Mutsumi Nakamura, Neeraj Varshney, and Chitta Baral. Multi-logieval: Towards evaluating multi-step logical reasoning ability of large language models, arXiv preprint arXiv:2406. 17/69, 2024.
[36] Hyun Ryu, Gyeongman Kim, Hyemin Lee, and Eunho Yang. Divide and translate: Compositional first- order logic translation and verification for complex logical reasoning. arXiv preprint arXiv:2410.08047, 2024.
[37] Damien Sileo. Scaling synthetic logical reasoning datasets with context-sensiticn declarative grammars. arXiv preprint arXiv:2406. 1005,, 2024.
[38] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed H Chi, Denny Zhou, et al. Challenging big- big-bench tasks and whether chain-of-thougie can solve them. arXiv preprint 2rX.v:221000926.. 2022.
[39] Oyvind Tafjord, Bhavana Dalvi Mishra, and Peter Clark. Proofwriter: Generating implications, proofs, and abductive statements over natural language. arXiv preprint arXiv:2012.13048, 2020.
[40] Fumiya Uchiyama, Takeshi Kojima, Andrew Gambardella, Qi Cao. Yusuke Iwasawa, and Yutaka Matsuo. Which programming language and what features at pre-training stage affect downstream logical inference performance? In EMNLP, 2024.
[41] Yuxuan Wan, Wenxuan Wang, Yiliu Yang. Youliang Yuan, Jen-tse Huang. Pinjia He, Wenxiang Jiao, and Michael Lyu. Logicasker: Evaluating and improving the logical reasoning ability of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2124-2155, 2024.
[42] Yuxuan Wan, Wenxuan Wang, Yiliu Yang. Youliang Yuan, Jen-tse Huang, Pinjia He, Wenxiang Jiao, and Michael R Lyu. A & b== b & Triggering logical reasoning failures in large language models. arXiv preprint or X4v:2401.00757, 2024.
[43] Binwu Wang. Pengkun Wang. Wei Xu, Xu Wang. Yudong Zhang, Kun Wang, and Yang Wang. Kill two birds with one stone: Rethinking data augmentation for deep long-tailed learning. The Twelfth International Conference on Learning Representations, 2024.
[44] Pengkun Wang, Zhe Zhao, HaiBin Wen, Fanfu Wang, Binwu Wang, Qingfu Zhang, and Yang Wang. Lim- autoda: I Large language model-driven automatic data augmentation for long-tailed problems. Advances in Neural Information Processing Systems, 17774915-64941 2024.
[45] Siyuan Wang, Zhongyu Wei, Yejin Choi, and Xiang Ren. Can llms reason with rules? logic scaffolding for stress-testing and improving lims. arXiv preprint arXiv:2402. 1442, 2024.
[46] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
[47] Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi, and Ravi Kumar. On memorization of large language models in logical reasoning. arXiv preprint a:Xiv:2410.23123, 2024.
[48] Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shich. Monte carlo tree search boosts reasoning via iterative preference learning. arXiv preprint arXiv:2405.0045/ 2024.
[49] Jundong Xu, Hao Fei, Liangmiigg Pan, Qian Liu, Mong-Li Lee, and Wynne Hsu. Faithful logical reasoning via symbolic chain-of-thought. arXiv preprint arXiv:2405. 200..8877.83 2024.
[50] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng. Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, et al. Qwen2. technical report. arXiv preprint 15X1v:2412.15155, 2024.
[51] Tian Ye, Zicheng Xu. Yuanzhi Li, and Zeyuan Allen Zhu. Physics of language models: Part 2.1, grade- school math and the hidden reasoning process. arXiv preprint arXiv:240.220311 2024.
13

---

[52] Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong. and Jie Tang. Rest-mets"; Lim self-training via process reward guided tree search. arXiv preprint 2Xix:2406.03816 2024.
[53] Honghua Zhang, Liunian Harold Li, Tao Meng, Kai-Wei Chang, and Guy Van Den Broeck. On the paradox of learning reason from data. In Proceedings of the Shiroy- International Joint Conference Artificial Intelligence, pages 3365-3373, 2023.
[54] Yifan Zhang, Yang Yuan, and Andrew Chi-Chih Yao. On the diagram of thought. arXiv preprint arXiv:2409. 10038, 2024.
[55] Wanjun Zhong, Siyuan Wang, Duyu Tang. Zenan Xu, Daya Guo, Jiahai Wang, Jian Yin, Ming Zhou, and Nan Duan, Ar-lsat: Investigating analytical reasoning of text, arXiv preprint urXiv:2104.0633 065000 2021.
14

---

Impact Statement, Limitations and Future Work
This paper follows existing research on logical reasoning data synthesis and constructs multi- hop complex logical reasoning dataset for model training, which helps LLMs learn advanced and generalizable logical reasoning capabilities. Furthermore, our synthesized data can be used to precisely evaluate the model's multi-step logical reasoning ability, contributing to research on enhancing models' logical reasoning capabilities. Moreover, our work does not involve human or animal experiments, so the ethical impacts and expected societal implications are those that are well established.
This paper presents work whose goal is to advance the field of logical data synthesis. There are many potential socictal consequences of our work, none of which we feel must be specifically highlighted here.
As for the limitations of LogicTree, while it significantly enhances the logical reasoning abilities of LLMs by synthesizing logical datasets, cannot be integrated with other forms of reasoning. Exploring how to incorporate other reasoning paradigms, such as commonsense reasoning, into logical reasoning data to enhance the comprehensive reasoning abilities of LLMs will be the focus of our future work.
A More Related Works
A.1 Prompt-Based Methods for Enhancing Logical Reasoning
Leveraging large language models (LLMs) for complex logical reasoning problems has been a key focus of recent research. Many efforts 12 25 focus designing more effective prompting strategies to help LLMs complete complex logical reasoning tasks. For example, Chain-of-
(C6 guides LLMs to reason step by step nd output the final answer based on the reasoning process. In order to better simulate rational logical thought processes, (author?) [54] proposed Diagram-of-Thouge (DoT) that models iterative logical reasoning in LLMs the construction of a directed acyelic graph (DAG). Other approaches involve prompting LLMs to first translate natural language into symbolic language, which is then used to facilitate logical reasoning. Symbolic Chain-of- ofoThhtthtttt (SymbCoT) [49] converts input from natural language into symbolic representation and subsequently constructs step-by-step reasoning plan based on symbolic logic rules. Logic-of-Thouu Thought [33 employs LLMs to extract logical expressions and applies logical reasoning rules to derive extended logical expressions, which are then translated back into natural language support subsequent reasoning.
Additionally, some research integrates LLMs with external tools, such as solvers, to enhance reasoning capabilities 26 (2). These approaches first leverage LLMs convert natural language input into a symbolic form (such as logic programming languages, first-order logic, constraint satisfaction problems, Boolean satisfiability formulations) compatible with external solvers, which are then used to perform logical reasoning and yield the desired result.
Although prompt-based methods can effectively leverage the potential of large language models (LLMs), their performance remains fundamentally constrained by the models' inherent reasoning capabilities. This limitation is particularly evident in smaller models, which often produce hal- lucinations and logical errors during inference. On the other hand, approaches that incorporate external solvers primarily exploit the model's ability to translate natural language into symbolic representations, rather than enhancing its intrinsic reasoning ability. In contrast, our approach focuses on synthesizing large amount of high-quality logical reasoning data from training perspective, which helps enhance the inherent reasoning capabilities of L.LMs.
A.2 Reasoning Data Synthesis
An increasing number of studies have focused on synthesizing complex tasks and high-quality rea- soning processes [43] 22), which are utilized for supervised fine-tuning or preference alignment training 17]. For example, the 01 series of studies leverages reinforcement learning and Monte Carlo Tree Search(MCTS)) that enable LLMS to optimize reasoning processes autonomously. Benefit- ing from the extensive tree search space and the guidance of an excellent Process Reward Model, these methods can generate high-quality reasoning processes 10 48). However, this approach incurs high computational costs and long search times, while generation efficiency is particularly
15

---

crucial in settings|II In contrast, our method leverages rigorous symbolic logic trees to guide the LLM, offering lower costs and higher correctness.
B More Details about Experiments
B.1 Information of Logical Reasoning Benchmarks
In this study, we selected six representative benchmarks to evaluate the reasoning capabilities of the models. These benchmarks encompass various aspects of logical reasoning, diverse contexts, and difficulty levels, making them suitable for comprehensive evaluation. As shown in Table 4 we provide brief explanations of each benchmark.
Table 4: Information of Logical Reasoning Benchmarks used in our experiments
Benchmark
Data Count
Explanation
LogicBench
160
LogicBench is natural language dataset designed to evaluate the logical reasoning abilities LLMs. In this work, we use the .MQAA (Multiple Choice Question-Answeriin subset of LogicBench(Eval), which focuses on tasks that apply a single inference rule.
LogiQA
1527 LogiQA 2.0 is an enhanced dataset for evaluating logical rea- soning in natural language understanding tasks. We use the main part of LogiQA, which includes multiple-choice reading comprehension questions.
203
FOLIO is an expert-crafted dataset for evaluating natural lan- guage reasoning with first-order logic. It includes logically complex examples presented in natural language and their for- mal FOL representations. idation data, leveraging its dual-format structure precisely assess models' ability to interpret and reason with formal logi- cal constructs.
FOLIO
our experiment, we use val-
BBH
1187
BIG-Bench Hard (BBH) subset of 23 challenging tasks from the BIG-Bench benchmark, focusing advanced rea- soning. For our experiment, we use three tasks: Causal Judg- ment, evaluating causal reasoning in stories; Formal Fallacies Syllogisms Negation, testing logical consistency in argument schemes; and Logical Deduction, assessing sequence deduction skills.
AGIEval
740
AGIEval is benehmark assessing foundation models' abilities through tasks from high-standard exams. For our experiment, we use the LSAT (Law School Admission Test) tasks, which evaluate logical reasoning, reading comprehension, and ana- lytical reasoning. These tasks challenge models to analyze complex information and draw accurate conclusions, providing valuable assessment of their capabilities in legal reasoning and analysis.
Multi-LogiBval 525
Multi- Multi-LogiEval a dataset designed to evaluate LLMs' multi- step reasoning abilities across various logic types. This focus al- lows for precise evaluation of L.MSs' capabilities in handling logical constructs while maintaining manageable complexity.
B.2 Experiment Setups
This section details our experimental design, covering model selection and the specific training configurations. To rigorously validate the effectiveness LogicTree, we conduct extensive exper- iments using diverse set of prominent open-source models spanning various families and scales.
16

---

We utilize models from the Llama-3.1, Mistral-v0.3, Qwen2.5, and Deepseek- Ri-ksstilD families, with parameter scales ranging from 1.5B to 70B. We employ two distinct tinettuning strategies: full fine-tuning is applied to amaller-scale models (under 8B), while the larger 70B model utilizes LoRA fine-tuning. I.amm-3.1 1-8B, Mistral-7B-v0.3 and Qwen2.5-7B are both trained with learning rate of le 6. Qwen2.5-1.5B and Qwen2.5-3B are trained with learning rate of 3e 6. Llama-3.1-70B, due to its LoRA fine-tuning method, is trained with a higher learning rate of 2e 5. The training utilizes a maximum context length of 4096 tokens, global batch size of 128, and is conducted for 3 epochs. DeepSpeed with gradient checkpointing and BF16 precision used for efficient memory usage. The learning rate scheduler follows cosine schedule, and the warmup ratio is set to 0.03.
B.3 More results of Different Models
To comprehensively validate the effectiveness of LogicTrree, We advanced reasoning models such as PrepSeee--1--iisill---mme
and DeepSeek-RI-De -iil---iilll--we-----7 As shown in the table 5 LogicTree consistently improves the performance of DeepSeek-R -i-----iil--isllmmaame
and DeepSeek- RI-Distill-Qwen-7B across multiple benchmarks, with average gains of 6.5 % and 4.9 %, respectively, Notably, on logic-intensive benchmarks such as the ML series, the improvements reach up 13.3 % and 10.9 %. These results further confirm the effectiveness of LogicTree in enhancing the logical reasoning capabilities of LLMs.
Table 5: Results of DeepSeek-RI RI--sstilll series models across multiple logical reasoning benchmarks.
Datasct
Model
Avg
LogieB.
LogIQA
FOLIO
AGIEval-LR AGIEval-AR ML-DI
ML-D2
ML-D3 ML-D4 ML-DS
DeepSeek-RU PrepSeek-RI-Dssill--e
Vanilla
86.8
52.4
61.2
56.8
32.6
70.0
57.0
60.0
55.0
44.4
57.6
TrgicTree
90.6
53.4
63.4
62.4
34.4
80.9
67.6
62.5
56.6
53.3
62.5
A
+3.8
1.0
+2.2
+5.6
+1.8
+10.9
+10.6
+2.5
+1.6
+4.9
Vanilla
83.1
53.4
53.1
53.3
30.4
78.0
67.1
60.0
49.2
46.6
57.4
TrogicTree
91.2
54.9
57.8
57.3
33.9
86.3
73.3
66.0
62.5
56.6
63.9
A
+8.1
+1.5
+4.7
+4.0
+3.5
+7.3
+6.2
+6.0
+13.3
+10.0
46.5
In addition,we we conduct additional experiments using Qwen2.5-1.5B and Qwen2.5-3B to further eval- uate the effectiveness of Logic'Tree smaller language models. As shown in the Table Logic'Tree effectively enhances the logical reasoning capabilities of smaller models such as Qwen2.5-1.50 and Qwen2.5-3B, achieving consistent and substantial improvements across multiple benchmarks. Specifically, Qwen2.5-1155 1.5B achieves an average accuracy gain of 7.7 %. and Qwen2.5-3B improves by 9.3 %. demonstrating that our synthesized data remains effective even for smaller-scale models.
Table 6: Results of small models across multiple logical reasoning benchmarks.
Dataset
Model
Avg
LogicBench LegiQA2.0 FOLIO AGIEval-IRR AGIEval-AR MI.DI ML-D2 MI.D3
ML-D4 MI.DS
Qwen2.5 5-1.58
Vanilla 65.0
43,9
48.5
39.5
17.4
65.6
53.3
48.8
43.3
31.1
45.8
+LogieTree
74.4
48.4
53.4
42.4
21.3
80.9
56.2
54.8
54.2
44.4
53.5
Qwen2.5.38
Vanilla
20.9
74.5
46.5
45.0
+LogicTree 83.9
56.2
61.2
55.7
24.4
86.3
67.6
59.3
58.3
50.0
60.5
B.4 More Results on Multi-LogiEEva
We have further conducted additional experiments on multi-step reasoning(sec4.33 using LLaMA3.1 70B to validate the robustness of our method. Although the 70B model shows some performance improvement over the 8B model various reasoning depths, it still faces decline in accuracy as the number of reasoning steps increases. When the reasoning depth reaches 5, the model's 's accuracy is only 44%. Our method significantly enhances the model's logical reasoning accuracy across different reasoning depths, demonstrating its superiority. As shown in the table 7 the 1..MA311-700 model achieved notable performance, even surpassing the vanilla model and baseline methods, which highlights the scalability of our approach.
17

---

Table 7: Results of Llama-3.1-70B on Multi-Logii
Model
Method
d1
d2
d3
d4
d5
vanilla
86.67
61.90
54.81
50.83
33.33
PARARULE
86.67
71.43
62.22
57.50
53.33
Llama-3.1-8B
logicAsker
91.66
58.09
57.03
56.66
57.77
FLD
86.67
72.31
61.85
56.47
38.89
LogicTrre
94.17
80.00
74.81
66.67
61.11
vanilla
76.67
54.29
51.85
37.50
26.67
PARARULE
81.67
73.33
57.04
40.00
37.78
Mistral-7B-v0.3
logicAsker
81.67
67.62
68.89
53.33
54.44
FLDx2
80.00
72.38
63.70
47.50
46.67
LogicTrree
93.30
78.10
71.85
66.67
57.78
vanilla
73.33
68.57
52.59
43.33
24.44
PARARULE
77.50
71.43
52.59
47.50
31.11
Qwen2.5-7B
logicAsker
75.83
71.43
54.07
45.00
28.89
FLDxx2
73.33
70.48
54.07
39.17
31.11
LogicTree
86.67
73.33
56.30
52.50
48.89
vanilla
90.8
64.7
58.2
61.7
44.4
PARARULE
93.3
69.5
59.3
63.3
56.7
Llama-3.1-70B
LogicAsker
93.3
69.5
60.0
65.8
46.7
FLD x2
90.8
66.7
63.9
61.7
46.7
LogicTree 96.7
73.3
76.3
74.2
60.0
B.5
Results on Quantitative Evaluation of Synthetic Data
Table 8: Evaluation of logical consistency of synthetic data across multiple models
Model
Consistency
GPT-4o
97.6%
Deepseek-RI
96.8%
Logical Consistency To minimize instantiation errors much as possible, we designed two- stage prompting strategy to guide the LLM in instantiating logical reasoning trees. Through this process, the LLM does not need to perform reasoning on own; it only needs perform symbolic translation. Additionally, we employed a systematic verification method to filter out errors and filtered out 8.73% of erroneous data, thereby improving the overall data quality. We also conducted additional experiments to evaluate the logical consistency of the filtered instantiated data. Specifically, we prompted multiple LLMs to independently assess whether the logical expressions and their corresponding natural language statements were logically consistent. As shown in Table our data maintained high level of logical consistency across evaluations by several state-of off-the-an art models.
The prompt used for evaluation with LLMs is follows:
You are an expert in logical reasoning and formal logic. Based on the correspondence between entities and logical symbols, your task evaluate the consistency between the given natural language statements and corresponding logical expressions. Please assess whether the logical expressions accurately translate the logical semantics of the natural language statements.
Realism Contextual Richness We conducted additional validation experiments to demonstrate the superiority of our synthesized data in terms of realism and contextual richness. Specifically, we employed multiple L.MMs to evaluate synthesized data and compared the results with other methods. The evaluation metries are defined follows:
18

---

Meitedss
PARARULE
Beebba Contectuaa Rehnwes (0-30)
Luggc
Reailam
Diversity
Figure 3: Comparison of Different Methods Across Realism, Contextual Richness, and Diversity. The red bars represent LogicTree, which significantly outperforms other baseline methods.
Assess whether the data presents a realistic, plausible, and logically consistent real-world scenario.
Realism:
2. Contextual richness: Assess whether the data includes diverse and detailed elements that enrich the scenario.
Notably, due to the rich world knowledge and strong language capabilities of LLMs, Logic'rre achieved the highest score for realism and contextual richness in Figure LrgieoTreggo synthetic reasoning data in concrete and context-rich real-world scenarios that incorporate semantically relevant entities events, capable reflecting the complexity and diversity of real-world reasoning tasks. The prompts used for evaluation with LLMs are as follows:
You are an expert in data evaluation, contextual analysis, and scenario validation. Your task is to evaluate the contextual richness of the generated data, assigning a single score out of 10 to reflect the overall quality.
The evaluation should consider the following criteria:
Contextual Depth: Assess whether the data includes diverse and detailed elements that enrich the scenario.
1.
2. Variety of Information: Determine whether the generated data presents broad range of relevant contextual elements.
Scoring Guidelines:
10: Very contextually rich with wide variety of relevant information. Generally rich in context, with sufficient variety and depth, though slightly lacking in some aspects. 8: Moderately rich with noticeable diversity, certain areas could be expanded enhance contextual richness. 6: Provides basic context with some variety. 5: Limited
contextual richness, with minimal diversity or detailed elements. Poorly developed with very few diverse contextual elements, resulting in shallow scenario.
Lacks any meaningful contextual richness, offering almost no variety depth.
19

---

You are an expert in You are an expert in data evaluation, logical reasoning, and real-world scenario validation. Your task is to evaluate the realism of the generated data, assigning a single score out of 10 to reflect the overall quality. The evaluation should consider the following criteria:
1. Realism: Assess whether the data presents realistic and plausible
real-world scenario.
2. Logical Consistency: Determine whether the generated data maintains internal logical coherence.
Scoring Guidelines:
10: Highly realistic and plausible. 9: Mostly realistic and consistent with minor deviations that not affect overall plausibility. 8: Generally realistic and coherent with slight inconsistencies or plausible.
Somewhat realistic but with inconsistencies that slightly affect coherence plausibility. but with noticeable flaws that slightly impact plausibility and background consistency.. 5: Lacks realism and consistency, with major flaws in logical coherence and background alignment. 3: Highly unrealistic and implausible, with multiple errors and misalignments. 1: Completely unrealistic and logically incoherent.
6: Reasonably realistic
Soiiing bos povincige
Figure 4: Statistical analysis of the diversity of the synthesized scenarios. We summarized over 50 common themes and prompted the LLM to instantiate multiple real-world scenarios for each logical reasoning tree.
Diversity Scenarios Our goal in introducing instantiation diversity is to enable the model to truly lear logical reasoning rules rather than simply memorizing implicit relationships between specific contents. To achieve this, we summarized over 50 common themes and prompted the LLM to instantiate multiple real-world scenarios for each logical reasoning tree. We also provided statistical aFalyse 4 of the diversity of the synthesized scenarios to demonstrate the effectiveness of this approach.
20

---

C More Details about Logical Reasoning Tree Generation
C.1 Logical Reasoning Tree Generation Process
Logical Rules
Applicable Rules Based on Structural Pattern Matching
Syllagism
(F + G)iG + +.)
syllogism
Instantiation
H
Modus Ponens
(AVB) + G
(+(
y J - 5
(AV B) - (-C)
Backward
Deduction
(A (AVB) G
G (-C)
-AVB)-(-0)
(-C)
C
AVB
VB
C
Figure 5: More Details about Logical Reasoning Tree Generation
To clarify our computational cost, we analyzed the number of LLM output tokens consumed when generating each data instance. Specifically, assuming that each reasoning step consumes T output tokens, the computational cost generating a data instance with reasoning steps m is as follows for each method. We define Instantiation Cost and Verification Cost.
1.
For the Instantiation 1ost: The first call to LLM populates the abstract logical symbols with concrete natural language entities. We denote the number of output tokens consumed in this step as c. 2) The second call translates the sequence reasoning steps into natural language. We assume that translating each individual reasoning step requires average of output tokens7. The computational cost of generating a data instance with reasoning steps is:
Costinet m - T +'+
2. For the Verification Cost: We assume that the complexity of verifying a single step is comparable that of translating during instantiation. Consequently, the computational cost for verifying data instance with reasoning steps is:
Coster m
As shown in the table 9 (the data in the parentheses), the ratio of synthesized tokens to consumed mf tokens for our method is 2mT++c (close to 1/2, in our data T = 79 + 14, 114 + 44), which significantly lower than the cost of other methods. Our framework's guarantee of correctness eliminates the need for multiple LLM calls. Consequently, its generation cost is entirely acceptable in comparison with other approaches.
C.2 Statistical Information of Key Properties for logical synthetic data
In order to construct logical reasoning tree, previous methods typically concatenate rules iteratively from predefined set and randomly replace atomic formulas with complex ones, resulting in an uncontrollable generation process. Moreover, rules are only concatenated when the conclusion of one
21

---

Table 9: A comparison of the generation costs for different LLM-based data synthesis methods.
Method
V-STaR
ALPHALLM
LogicTree
Number of LLM invocations
k
mbk
3
Number of consumed LLM output tokens
mkt
mbb (oollo))) ) + mT
2mT +
Number of synthesized tokens
mī
mT
mT
Explanation
Where & is the number of candidate reasoning trajectories.
Where is the number of candidate reasoning nodes per step, k is the number of rollout simulations for each node, and Toutluu is the additional tokens consumed during the rollout process.
where e denotes the tokens consumed during the first invocation to generate the instantiated entities.
matches the premise another, which restricts the diversity and complexity of reasoning patterns preventing the integration of propositional and first-order logic rules).
In contrast, we designed a backward deduction method based on structural pattern matching of formulas, which does not require the formula to be identical to the conclusion of the logic rule; instead, only compares whether the structural patterns of their are matched. In this way, we generate multi-step logical reasoning trees that incorporate diverse and complex reasoning patterns, encompassing both propositional logic and first-order logic. Table [T] we provide additional details about our synthetic data and compare it with other methods. And in table [T] we present supplementary statistical information to further characterize our dataset.
Table 10:
The comparison of synthetic logic corpora, which focuses several key characteristics: the number of logical rules, the reasoning depth, the symbolic-to-natiee language translation method, and the instantiation approach.
Logic Rules
Reasoning Steps
Translation
Instantiation
RuleTaker PARARULE
2
1-5
Template
Random Entities
2
1-5
Template
Random Entities
FLD
13
1-8
Template
WorldNet
FLDx2
Rs 50
1-8
Template
WorldNet
LogicTree
190
1-15
L.M--based
Realistic Scenario
Table 11: Statistical information for logic trees with 5 to 8 reasoning steps, including (1) the average maximum depth of the trees, (2) the average number of nodes per tree, (3) the average number of distinct rules applied, well (4) the proportion of first-order logic rules and propositional logic rules used in each tree.
Step
Number of Maximum Nodes
Number of Distinct rules
Proportion of Fol
Proportion of Prop
Depth
5
11.46 + 1.19
3.68 + 1.42
4.14 + 0.52
31.42
68.58
6
13.46 1.01
5.04 + 0.32
4.75 + 0.81
30.45
69.55
7
15.78 1.81 18.30
5.61 + 0.56
5.29 + 0.72
29.94
70.06
8
2.01
6.04 0.76
5.67 0.97
28.82
71.18
22

---

C.3 Logical Rules used in LogicTrree
This section presents the logical rules employed the LogicTTree generation process. These rules are fundamental to automated reasoning and inference generation, serving as the foundation for constructing structured logical trees. Drawn from both First-order Logic and Propositional Logic, the applied rules encompass variety of logical inference patterns, including Modus Ponens, Hypothetical Syllogism, Disjunctive Syllogism, and Universal Instantiation. Table presents the symbolic form and natural language explanation for selection of common logical rules.
Table 12: Some Logical Rules Used in LogicTrree Generation Process and Their Explanations
Rule Name
Rule Symbol
Explanation
First-order Logic
MP
(Vx(p(æ) - A s(a)
Modus Ponens: Universal elimination combined with conjunction provides the result q(a).
(Va(p(>) - -(=))) ppa)
Modus Tollens: From universal climination and negation, derives -p(a).
(Val(p(z) q())) (4(x)
HS
Hypothetical Syllogism: Nested implications for quantified variables result the conditional p(a) + r(a).
r())))) (p(a) r(a))
DS
(V(() A -p(a)) g(a)
Disjunctive Syllogism:
Disjunction with univer- sal quantification simplifies to g(a) if pp(a) is given.
UI
vx(p(=)) + p(a)
From the universal quan- tifier Va((((r)), deduce the specific instance p(a).
Universal Instantiation:
Propositional Logic
MP
((P - (00-)) A))
Modus Ponens: If
and p. then
DS
((DV)) A -p) 1g
Disjunctive Syllogism: If V and -p, then Modus Tollens: If
MT
((p 9) A -2) + >p
and -q, then -p. Hypothetical Syllogism: If p q and then
HS
(lp 2) + r)) + r)
MI
(p (p+*g) 4) + (-)Vg)
Material Implication: Expresses conditional disjunction.
DMT
-ppA0) + V
De Morgan's Theorem: Simplifies negation of conjunctions.
((p g)((r + 3) (gVs)
Constructive Dilemma: Combines conditional and disjunctive reasoning.
((p + A (F- -))
DD
Destructive Dilemma: one or more results fail, one or more premises fail.
(PV
((p + 9) / (r + 8) (pv -5)
BD
Bipolar Dilemma: Variation of dilemma reason- ing combining conditions.
(g -))
23

---

D Details about Prompts for Scenario Instantiation
This section discusses the prompts used during the reasoning scenario instantiation step, detailing their structure and role in guiding the model to produce relevant and coherent outputs for logic trees.
D.1 Logical Tree Instantiation
As shown in Figure logical tree instantiation prompt primarily consists of five components: 1. Role-play: Setting the context for the model to assume relevant role. 2. Instantiation instruction: Guiding the model instantiate logical symbols using appropriate real-world scenarios. Logical expression translation instruction: Assisting the model in understanding key logical relationships by providing in-context learning examples for translation.
4. Diversity control: Selecting different instantiation scenarios to ensure diversity. 5. Specific instantiation in-context learning examples: Offering concrete examples to further guide the instantiation process.
You are highly skilled logic analyst with expertise in understanding and interpreting complex logical relationships.
Please deeply understand the logical rules between the propositions. Note the meanings of the logical symbols:
(NOT): Negation, indicates the proposition is not true.
''>' (IMPLiES):lmplies, indicates that there inferential relationship between the two propositions.
-I'' (OR): Logical disjunction, indicates that at least of the propositions either side
(AND): Logical conjunction, indicates that both propositions either side
'Vx' (FOR ALL): Universal quantifier, indicates that statement true for all values of
While satisfying the logical rules, replace the symbolic propositions with real life complex events from the relevant field. Please think step by step logic rules.
that the replaced events naturally satisfy the
Use the corresponding events to explain each logic rule expression in natural language. Please follow the priority order in logical expression, think step step, ensure the translation is accurate clear. example:
i: PL=("((-00)&0))---ff P1 true, either PO is true, ar Q is false, or both.
ii: (*( Rf both Q and RO are false, then P2 must be true. false,
least of Q iv: ((118)))=((QQ)==)))--- both P1 and are true, then [if Q false, must also be true.]. (vx(P4())))--or all P4(x) is true.
Generate examples from multiple domains [[domain)), We hope you can leverage relevant entities this field to construct the aforementioned complex events.
example1 (example1)
example2 (example2)
example3
(example3)
Figure 6: Prompt for Logical Tree Instantiation
24

---

D.2 Reasoning Process Generation
As shown in Figure our reasoning process generation prompt primarily consists five components: 1. Role-play:
Setting the context for the model to assume relevant role. 2. Understanding of logical relations: requires recognizing logical operators, understanding how propositions are structured, and determining how they interact based formal logic rules. 3. Step-by-Step Reasoning Process Generation: In this stage, a structured sequence of logical steps is generated, following formal inference rules. Each step should be clearly justified to show how the conclusion is logically derived from the premises. derived from the reasoning process and offers concise explanation of why the conclusion is valid. Specific reasoning process generation learning examples: Offering concrete examples to further guide the instantiation process.
Conclusion and Answer Explanation: This part provides the final outcome
You are highly skilled logic analyst with expertise in understanding and interpreting complex logical relationships.
The following is a logical reasoning context and some conclusions. For each conclusion the truth value can be Yes, No or Uncertain. And We have extracted the main event entities and logic expressions from the context and conclusions. Note the meanings of the logical symbols: (NOT): Negation, indicates proposition is
(IMpLiss)::mplies, indicates that there inferential relationship between two propasitions.
'I' (OR): Logical disjunction, indicates that at least one of the propositions on either side is true.
8 (AND): Logical conjunction, indicates that both propositions either side true. for values of
'VX'' (FOR ALL): Universal quantifier, indicates that
Please refer to the following reasoning steps and the extracted entities provide detailed reasoning explanation natural language. For each step reasoning, we provide the given premises the conclusions that can be deduced. Please carefully explain each step in detail.
each conclusion, refer the reasoning process generated above and provide an explanation that aligns with the corresponding answer.
example1 (example1)
-ex.mple2 (example2)
-example3-... (example3)
Figure 7: Prompt for Reasoning Process Generation
25

---

E Examples of Synthetic Logical Reasoning data
Logical Reasoning Tree
Synthetic data
LogicTree
Others
Xx (xx(())
No Relevance
(-R(x))
banana
1.Wx (-R(x)) robot.
Everyone is not
that speak French.
R
2.P + If unicarn won the
2.P > T: somoeene presses the emergency button, then the alarm sound.
lottery, then the moon turned
PV (W(or))
into cheese.
3.T + Q: If the moon turned into cheese, then socks disappeared worldwide.
the sounds, the main door will locked.
4.P R(a): Either pressed the emergency button, Max robot.
4.P R(a) Either unicor won the lottery, Albert banana that French.
Figure 8: The logical reasoning trees and the corresponding synthetic data presented, with the left side generated by LogicTree and the right side produced using template-based random substitution method. It can be observed that the data synthesized by LogicTree maintains coherent contextual semantic relationship, whereas the data generated by other methods is either unrelated or even contradictory.
P5
Q1]T
vx(P3(x))
TO>P1
Q1>P2
P5>P4
P3(a)
P2
P4
PP2
(P3(1)))))
P1
P1>Q0
Q0
Figure 9: An example of complex logical reasoning tree, where green blocks implying the leaves to generate scenarioes
26

---

Entities:
P1: Emergency shelters were opened in time.
PP2: Casualties occurred.
P3(():: x is located a high-risk seismic zone.
P4: Local authorities activated the emergency response protocol.
PP5 A magnitude 6.0 or greater earthquake was detected.
-Q0: People were safely evacuated.
QQ1: Warning sirens activated.
TO: The national emergency broadcast was triggered.
.a: City A
Rules:
rule1: ((P3(a) P4) - (P1 - Q0)) City A is in high-risk seismic zone and local authorities
activated the emergency response protocol, then if emergency shelters were opened in time, people were safely evacuated.
rule2: vx(P3(x)) All regions located high-risk seismic zones.
rule3: (P5 - P4) a magnitude or greater earthquake was detected, then the emergency response protocol was activated.
rule5: (Q1 - P2) If warning sirens were activated, then casualties occurred.
rule6: (TO - P1) the national emergency broadcast was triggered, then emergency shelters were
opened in time.
rule7: (Q1 TO) Either warning sirens were activated, or the national emergency broadcast was triggered.
rule8: ~P2 No casualties occurred.
Reasoning process:
Step1: From rule5 (Q1 - P2) [if warning sirens were activated, then casualties occurred]. rule7 (Q1 v TO) (either warning sirens were activated or the national emergency broadcast was triggered], and rule6 (TO - P1) [if the national emergency broadcast was triggered, then emergency shelters were opened in time].
deduce that (P2 P1) [either casualties occurred or emergency shelters
opened time].
Step2: According to rule2 (vx(P3(x)) [all cities are located in high-risk seismic zones], we can conclude that (P3(a)) [City A located in high-risk seismic zone].
Step3: From rule3 (P5 P4) [if magnitude 6.0 greater earthquake detected, then the emergency response protocol detected], we can conclude that (P4) [the emergency response protocol was activated].
activated] and assuming (P5) (a magnitude 6.0+ earthquake
Step4: According to rule1 ((P3(a) A P4) - (P1 - Q0)) [if City A in a high-risk seismic zone and the emergency response protocol was activated, then if emergency shelters were opened in time, people were safely evacuated], and from step2 (P3(a)) and step3 (P4), we deduce that (P1 Q0) [if emergency shelters were opened time, people Step5: From step1 (P2 v P1) [either casualties occurred emergency shelters were opened time] and rule8 (-P2) [no casualties occurred], we can conclude that (P1) [emergency shelters were opened in time].
safely evacuated].
Step6: From step4 (P1 Q0) [if emergency shelters opened in time, people safely evacuated] and step5 (P1) [emergency shelters opened in time], conclude that (Q0) (people were safely evacuated].
Figure 10: The Generated Entities, Rules and Reasoning Process of an Example
27

---

conclusiont: people were safely evacuated. answer: Yes
explanation: From step6, we deduced that people were safely evacuated (Q0). Therefore, the conclusion that people were safely evacuated is true.
conclusion2: emergency shelters were not opened in time
answer: No
explanation: From step5, deduced that emergency shelters were opened in time (P1). Therefore, the conclusion that emergency shelters were not opened in time false.
conclusion3: either casualties occurred or emergency shelters were opened in time answer: No
explanation: From step1, deduced that either casualties occurred or emergency shelters were opened time (P2 v P1). From step5, we know emergency shelters were opened time (P1), making the statement "It is not the case that either casualties occurred emergency shelters were opened in time" false.
conclusion4: it is not the case that if emergency shelters were opened in time, people were safely evacuated
answer: No
explanation: From step4, we deduced that if emergency shelters were opened in time, people were safely evacuated (P1 - Q0). Therefore, the conclusion that it is not the case that if emergency shelters were opened in time, people were safely evacuated is false.
Figure 11: The Generated Answers an Example
28

---

Instantiated Realistic Reasoning Scenario1
major grant and the selacted partacapant "a" provises
Logical Tree
comp servey respense, then research findings
joural.
(PI rS]: "itthe the sociology research project is funded by voluscoers, both. (+P3); research
grant, or research
(P1)
has not secures additiona!
xP4(a)
Sociology
volunteers.
(Ox()) (2(xx))) "Mar all participants 'x', eitter they introductory briefing, Late survey respense, beth."
xP3
(P(()[)) [P2(x)]
all: *Participant has not attended the introductory briefiing
Instantiated Realistic Reasoning Scenario2
(P16P2(aa)>00
typerparaneters aptimized and the nodel evaluated on test cataset "a', than
(CP) p2ta)) 001:
Machine Learning
oo
(P1 P311 "Either the model is hyperparaneters have been aptinszed, the
includes
step,
[wPal: "the model does not inctude a feature selection step.
Inode
node
set been nerea nermalized, or the motel eva evaluates test both. (<Pila)): "The
normalized."
Figure 12: An example of instantiating a logic tree for diverse scenarios
Logical Tree
Realistic Reasoning Scenario1
nox zoning aquiaation, then the city council has recetved the comitteess zoting proposal."" (R0 F11:"Ir proposal, then the new zantnng regulations have been onacted into
(P3(a)>58)
councti received
(P2ta) (-P1))s 'If the environnental inpact assessnent for the nov has comple lated,
(x(P3(x) [2(x))]
(-P3(a))
the Ma zoning regulations have not
Urban Planning
(ExIPSIx) P2tx) ]] hor all comuunity nembers, either the comuuiity envaroneental inpact assessment for the now park has been comp approved.
opinisn community survey the construction
budget
approved."
(RBxP1)
(-PP)))(-PP)
Instantiated Realistic Reasoning Scenario2
[-Q8)>RR -RB
(-P1))
((-08) 'T
deened financially
Project
viable, than the project soves the ing Legerntatioonn phase.
phase,
(-Q8)>P1
Approval
then has received rinal appreval from the beard,
(Pa1a) (w2lll:
decunestation, project nanage
the final
approval. froe the beard.
@o
comp leted their assigned tasks, or the project nanager
necessary decunettation
root node
Inode
leaf node
tasks, then the project is delayed.
(PSta) 'It
completed
(<581;
project
Figure 13: An example of instantiating a logic tree for diverse scenarios
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the introduction, we state our main contributions and provide extensive experiments to support these claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: In the final section of the article, we discuss the limitations of this work and future directions.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
30

---

Answer: [NA]
Justification:
This work primarily focuses on synthetic data and model training.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide detailed explanation of the methodology in the Section 3 and offer a comprehensive experimental setup in the Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path reproducing verifying the results. 5. Open access to data and code
31

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: we use open-source datasets, and the code used in this paper will be made publicly available upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https/cccppllligggddpeeeeeeeeeee Cooee///ddeeebmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We offer a comprehensive experimental setup in the Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We follow relevant works in this field to report experiment statistical signifi-
cance
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
32

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We offer a comprehensive experimental setup in the Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Our research fully complies with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss this part in detail in the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
33

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: the paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all of the original paper that produced the code package dataset.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
34

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
35

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: In the method section, detail the use of LLMs and provide specific prompts.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy hctppe///earrrpsccc/o/ooeeeeee/e /ctps/////oooeeeeee//////////// for what should or should not be described.
36


Reviews:
[{"review_id": "GdYXbqpw5I", "reviewer": "Reviewer_rqDh", "summary": "The authors introduce “LogicTree,” a framework for synthesizing multi-step logical reasoning datasets with a focus on complexity and real-world contextual instantiation. The approach uses structural pattern matching for backward deduction to generate logic trees governed by first-order logic rules, and then employs a two-stage LLM-based method to instantiate these symbolic trees into diverse, context-rich, natural language scenarios. Logical consistency is checked post-hoc to filter poor samples. Experiments across several logical reasoning benchmarks demonstrate that training LLMs with LogicTree data improves accuracy over baselines.", "questions": "* What are the practical, computational costs of large-scale synthetic data generation and verification using LLMs, and do these limit applicability for broader academic use?\n* How sensitive are the results to the number of scenario instantiations per logic tree? Is there an upper bound after which gains saturate or reverse?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n* Methodological soundness: The paper clearly articulates the LogicTree pipeline, which systematically combines symbolic backward deduction with LLM-based scenario instantiation. The logic tree generation based on structural pattern matching is well-grounded in formal logic and illustrated stepwise.\n* Comprehensive evaluation: The experiments benchmark the method across diverse challenging datasets and tasks, including ablation studies for scenario instantiation and process diversity. Results support the main claims with quantitative improvements.\n\n**Weaknesses**\n* LLM-based verification: The logic consistency verification relies on LLMs themselves to make sure consistency between instantiated text and logic formulas (Section 3.3). This may introduces a potential failure mode where systematic instantiation errors might go undetected.\n* Depth in real-world evaluation: While the benchmarks are diverse and the accuracy improvements are robust, there is insufficient qualitative analysis to illustrate how the reasoning behavior of LLMs undergoes qualitative changes after training with LogicTree. For instance, do the models reason more explicitly, avoid hallucinations, or perform better in out-of-domain real-world reasoning? Specifically, on which types of questions are the improvements most significant, and on which do they even decline? More interpretive or failure-case analyses would strengthen the argument that such improvements stem from generalization rather than mere memorization.", "was_revised": true, "final_justification": "The responses address most of my concerns about the LLM verification and more detailed analysis of the dataset. Detailed statistics here should be added to the paper to make it clearer for readers.", "rebuttal": "**We thank the reviewer for the insightful and valuable comments. We respond to each comment as follows and sincerely hope that our rebuttal could properly address your concerns. If so, we would deeply appreciate it if you could raise your score (Rating: 4: Borderline accept). If not, please let us know your further concerns, and we will continue actively responding to your comments and improving our submission**.\n## Weakness 1\n> LLM-based verification: ...This may introduces a potential failure mode where systematic instantiation errors might go undetected.\n\nThank you for your valuable comment. We acknowledge that using LLMs for verification inevitably introduces some detection errors. However, **our framework mitigates this by simplifying the verification task**. Instead of tasking the LLM with the complex evaluation of global logical consistency, **we decompose the problem into a series of localized verification steps**. In each step, the LLM simply verifies the translation between a logical expression and its natural language counterpart, a task at which state-of-the-art models excel [1,2]. **Once each step is correctly translated, the symbolic nature of the LogicTree inherently ensures the overall logical consistency**.\n\n**To further address the reviewer's concern, we manually evaluated 500 randomly selected samples and found an error rate of only 4.6% (23/500)**. These minor errors were primarily caused by the model using incorrect entities or misunderstanding certain logical relations within a single reasoning step.\n\nMoreover, we would like to emphasize that the purpose of our synthetic data is **not to teach the LLM to memorize specific instantiations of natural language statements, but rather to help it learn to apply underlying logical rules across diverse instantiation scenarios**. Given this objective, we contend that this small proportion of noise is acceptable.\n\n## Weakness 2\n> Depth in real-world evaluation: ...More interpretive or failure-case analyses would strengthen the argument that such improvements stem from generalization rather than mere memorization.\n\nWe sincerely appreciate your insightful observation that more in-depth qualitative analysis is necessary to substantiate our quantitative findings. We will include more examples of real test cases in the appendix of the final version. Here, we present a representative examples, obtained by **comparing the performance of Llama-3.1-8B-Vanilla and our LogicTree-trained model on the MultiLogic-Eval benchmark**.\n\n\n### Case 1\n```\nPremise 1 : If a biological sample is labeled \"High-Risk\" (P), then the handler must wear a Level-A protective suit (Q). (P → Q )\n\nPremise 2 : A review of today's lab footage shows that a researcher, Dr. Li, was not wearing a Level-A protective suit while handling sample S-007.(¬Q)\n\nConclusion to Evaluate: an it be determined that sample S-007 was labeled \"High-Risk\"? (¬P)\n```\n\nVanilla Model's Response: No\n\nLogicTree-trained Model's Response: Yes\n\n**The LogicTree-trained model correctly identifies and applies Modus Tollens (a core idea underlying the well-known method of proof by contradiction)**. It adheres closely to the given logical structure, avoiding distractions from irrelevant speculations. This indicates that the training process has effectively internalized fundamental deductive principles, allowing the model to reason reliably from premises rather than relying on superficial pattern matching.**earning this rule is instrumental in enabling the model to apply proof by contradiction—an essential technique in mathematical reasoning.**\n\n## Question 1\n> What are the practical, computational costs of large-scale synthetic data generation and verification using LLMs\n\nWe appreciate the reviewer's insightful comment regarding the importance of computational cost associated with synthetic data generation. Indeed, a detailed analysis of the generation cost is provided in Appendix.B.5. For the reviewer's convenience, we summarize the key findings below. To clarify our computational cost, we analyzed *the number of LLM output tokens* consumed when generating each data instance.\n\n* **Instantiation Cost**:  1) The first call to LLM populates the abstract logical symbols with concrete natural language entities. We denote the number of output tokens consumed in this step as $c$. 2) The second call translates the sequence of reasoning steps into natural language. We assume that translating each individual reasoning step requires an average of $T$ output tokens. The computational cost of generating a data instance with reasoning steps $m$ is:\n$$Cost_{inst}=m⋅T+c$$\n* **Verification Cost**: We assume that the complexity of verifying a single step is comparable to that of translating it during instantiation. Consequently, the computational cost for verifying a data instance with m reasoning steps is: \n$$Cost_{ver}=m⋅T$$\n\n||number of LLM invocations|Number of consumed LLM output tokens|Number of synthesized tokens|explanation\n|-|-|-|-|-|\n|V-STaR[1]|k|mkT|mT|k is the reasoning times.|\n|ALPHALLM[2]|mbk|$mbk(T+T_{rollout})$|mT|b is the reasoning times per step, k is the number of rollout simulations for each node, and $T_{rollout}$ is the additional tokens consumed during the rollout process.|\n|**LogicTree**|3|2mT+c|mT| |\n\nAs shown in the table, the ratio of **synthesized tokens to consumed tokens** for our method is $\\frac{mT}{2mT+c}$ (close to 1/2,in our data $T = 79_{\\pm14}$, $c = 114_{\\pm44}$),which is significantly lower than the cost of other methods.   Our framework's guarantee of correctness eliminates the need for multiple LLM calls. Consequently, its generation cost is entirely acceptable in comparison with other approaches.\n\n\n## Question 2\n> How sensitive are the results to the number of scenario instantiations per logic tree? Is there an upper bound after which gains saturate or reverse?\n\nWe sincerely thank the reviewer for the insightful comment. We would like to clarify that an initial analysis on this topic was provided in our ablation study in **Section 4.4** of the manuscript.\n\nIn that section, we instantiated each logical tree into a varying number of reasoning scenarios (from 1 to 3), with our main results using three scenarios. As presented in Table 2, we observed that increasing the number of scenarios led to a consistent performance improvement across most benchmarks (e.g., from 51.6% to 55.3%). This finding supports our conclusion that greater diversity in reasoning scenarios is beneficial for the model to learn genuinely generalizable logical reasoning skills.\n\nWe apologize for the omission and agree that exploring the upper bounds of this effect is valuable. To address this, we have conducted additional experiments for 4 and 5 instantiated scenarios. The complete set of results, including these new data points, is summarized below:\n\n\n|Model|LogicBench|LogiQA2.0|BBH|AGIEval-LR|AGIEval-AR|Avg|\n|-|-|-|-|-|-|-|\n|Llama3.1-8B|80.0|42.4|39.3|48.4|20.7|45.0|\n|num=1|83.1|51.2|49.6|54.1|23.9|51.6|\n|num=2|88.1|52.9|50.8|52.9|26.1|53.2|\n|num=3(used in LogicTree)|90.6|**53.5**|**53.2**|**56.1**|**26.5**|**55.3**|\n|num=4|92.1|52.7|**53.2**|**56.1**|25.7|55.2|\n|num=5|**94.5**|50.3|50.8|51.2|23.7|54.1|\n\n**As the results in the table indicate, the performance of Llama-3.1-8B peaks when using 3 or 4 instantiations**. However, upon further increasing the number to 5, we observe a slight decline in performance. Therefore, **taking into account both the computational overhead and the performance curve, using 3 instantiations offers the best cost-benefit trade-off and was chosen for our main experiments**.\n\nWe humbly hope our response has addressed your concerns. If you have any additional concerns or comments that we may have missed in our responses, we would be most grateful for any further feedback from you to help us further enhance our work.\n\n### Reference:\n[1] Faithful Logical Reasoning via Symbolic Chain-of-Thought. ACL, 2024\n\n[2] Divide and Translate: Compositional First-Order Logic Translation and Verification for Complex Logical Reasoning. ICLR, 2025"}, {"review_id": "lr6gd2O6VH", "reviewer": "Reviewer_sD1m", "summary": "The paper proposes a novel framework called LogicTree for synthesizing multi-step logical reasoning datasets. The method leverages first-order logic rules to generate complex reasoning trees and instantiates them using Large Language Models to produce natural language reasoning data with realistic scenarios. The paper conducts experiments across multiple benchmarks and demonstrates a good improvement of accuracy on complex logical reasoning tasks.", "questions": "1. Can you provide more detailed statistics on the generated datasets?\n2. How does the LLM-based instantiation process handle the hallucination issues of LLMs to ensure the correctness of the generated statements? \n3. Can you consider using more advanced large reasoning models as DeepSeek-R1-distill-8B, to further validate the effectiveness of the LogicTree framework in improving the reasoning abilities of LLMs?", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Paper Strengths:\n1. The proposed LogicTree framework is innovative in efficiently synthesizing multi-step logical reasoning datasets with complex reasoning patterns.\n2. The two-stage LLM-based instantiation technique for translating symbolic logical expressions into concrete statements with contextual significance is insightful. This approach enhances the realism and applicability of the generated datasets.\n3. The paper provides in-depth experimental results demonstrating the effectiveness of LogicTree in enhancing LLMs' reasoning abilities across various benchmarks and questions of different complexity levels.\n\n## Paper Weaknesses:\n1. Lack of details statistics on the generated datasets, such as the number of questions, unique reasoning patterns, complexity levels, and distribution of question types. This information is crucial for understanding the diversity and coverage of the synthesized datasets.\n2. The LLM-based instantiation purely relies on the LLMs' capabilities to generate natural language statements and inject factual knowledge. However, the LLMs suffer from hallucination issues, which may lead to the generation of incorrect or misleading statements with unreliable factual knowledge. The paper should provide more analysis on how the instantiation process handles such issues and ensures the correctness of the generated statements.\n3. The used baseline LLMs are just the vanilla LLMs without being specifically designed for intricate reasoning abilities. The paper should consider using more advanced large reasoning models, such as DeepSeek-R1-distill-8B, to further validate the effectiveness of the LogicTree framework in improving the reasoning abilities of LLMs.", "was_revised": true, "final_justification": "The responses of the author address my concerns about the data statistics, quality control, and performance improvement for advanced LRM. Thus, I update my score to accept.", "rebuttal": "Thank you for your valuable and constructive review. We sincerely appreciate your feedback. Below, we respond to each of your comments in detail and hope that our responses could properly address your concerns. If you find them satisfactory, we would be grateful if you could consider updating your score. Otherwise, please feel free to let us know any remaining questions or concerns, and we will be happy to continue the discussion.\n\n# Weakness 1 & Question 1\n> Lack of details statistics on the generated datasets, such as the number of questions, unique reasoning patterns, complexity levels, and distribution of question types. \n\nThank you for your valuable comments. We generated 5,000 symbolic logic trees with **depths from 2 to 15**, and instantiated each into **3 semantically diverse scenarios**, yielding 15,000 reasoning problems. After applying an automatic filtering process that discarded 8.73% of noisy or invalid samples, **the final dataset for LLM training contains **1.38k** high-quality, multi-step reasoning instances**. \n\nAn analysis of our synthetic data's properties and a comparison with other methods are presented in **Appendix C.3**. For the reviewer's convenience, we reproduce the key information below.\n||Logic Rules|Reasoning Steps|Translation|Instantiation|\n|-|-|-|-|-|\n|RuleTaker|2|1-5|Template|Random Entities|\n|PARARULE|2|1-5 |Template|Random Entities|\n|FLD|13|1-8 |Template|WorldNet |\n|FLD$_{\\times 2}$|$\\approx$ 50|1-8|Template|WorldNet|\n|**LogicTree**|**190**|**1-15**|**LLM-based**|**Realistic Scenario**|\n\nIn addition, the following table presents supplementary statistical information to further characterize our dataset. The table below presents statistical information for **logic trees with 5 to 8 reasoning steps**, including (1) the average maximum depth of the trees, (2) the average number of nodes per tree, (3) the average number of distinct rules applied, as well as (4) the proportion of first-order logic rules and propositional logic rules used in each tree.\n\n\n\n|Step|Number of Nodes|Maximum Depth|Number of Distinct rules|Proportion of Fol|Proportion of Prop|\n|-|-|-|-|-|-|\n|5|$11.46_{\\pm1.19}$|$3.68_{\\pm1.42}$|$4.14_{\\pm0.52}$|$31.42\\%$|$68.58\\%$|\n|6|$13.46_{\\pm1.01}$|$5.04_{\\pm0.32}$|$4.75_{\\pm0.81}$|$30.45\\%$|$69.55\\%$|\n|7|$15.78_{\\pm1.81}$|$5.61_{\\pm0.56}$|$5.29_{\\pm0.72}$|$29.94\\%$|$70.06\\%$|\n|8|$18.30_{\\pm2.01}$|$6.04_{\\pm0.76}$|$5.67_{\\pm0.97}$| $28.82\\%$|$71.18\\%$|\n\n\n# Weakness 2 & Question 2 \n> How does the LLM-based instantiation process handle the hallucination issues of LLMs to ensure the correctness of the generated statements?\n\nThank you for this insightful question. Ensuring the correctness of the generated statements is a critical component of our methodology. We address this concern from two key perspectives: (1) our proactive methodological design and (2) a systematic error verification process.\n* **Two-stage LLM-based instantiation**: Due to the inherent limitations of LLMs in handling complex logical reasoning, directly prompting them to generate realistic logical reasoning problems from symbolic logic trees may introduce errors and hallucinations. Therefore, to address this challenge, **we designed a two-stage prompting strategy to reduce the task's difficulty for the LLM**. The first stage prompts the model to assign factually and logically coherent entities to the symbolic variables in the leaf nodes. Subsequently, the second stage instructs the LLM to translate the symbolic reasoning process into natural language on a step-by-step basis. This decomposition ensures that when generating the natural language statements, **the LLM only needs to focus on the translation of individual logical expressions**, rather than grappling with the complex dependencies of the entire logic tree. This targeted approach effectively minimizes hallucinations and improves the fidelity of the generated natural language statements.\n* **Error verification process**：Furthermore, to minimize errors in the instantiated data, we implemented a systematic verification process. For each generated instance, **we employed an LLM-based verifier to check for logical fallacies and implausible natural language statements** by comparing them against their corresponding logical expressions. This process successfully **filtered out 8.73% of erroneous data**, thereby significantly enhancing the overall quality of the dataset.\n\nTo further address the reviewer's concern, we manually evaluated 500 randomly selected samples and found an error rate of only 4.6% (23/500). These minor errors were primarily caused by the model using incorrect entities or misunderstanding certain logical relations within a single reasoning step.\n\nMoreover, we would like to emphasize that the purpose of our synthetic data is **not to** teach the LLM to memorize specific instantiations of natural language statements, but rather to **help it learn to apply underlying logical rules across diverse instantiation scenarios**. Given this objective, we contend that this small proportion of noise is acceptable.\n\n\n# Weakness 3 & Question 3\n\n> Can you consider using more advanced large reasoning models as DeepSeek-R1-distill-8B, to further validate the effectiveness of the LogicTree framework in improving the reasoning abilities of LLMs?\n\nWe appreciate your suggestion to supplement our work with experiments involving more advanced reasoning models, as this would provide important evidence to further validate the effectiveness of our dataset. We further include experiments using advanced reasoning models such as DeepSeek-R1-Distill-Llama-8B and DeepSeek-R1-Distill-Qwen-7B to strengthen our evaluation.\n\n|DeepSeek-R1-Distill-Qwen-7B|LogicBench|LogiQA2.0|FOLIO|AGIEval-LR|AGIEval-AR|ML-D1|ML-D2|ML-D3|ML-D4|ML-D5|Avg|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|Vanilla|86.8|52.4|61.2|56.8|32.6|70.0|57.0|60.0|55.0|44.4|57.6|\n|+LogicTree|90.6(+3.8)|53.4(+1.0)|63.4(2.2)|62.4(5.6)|34.4(+1.8)|80.9(+10.9)|67.6(+10.6)|62.5(+2.5)|56.6(+1.6)|53.3(+9.9)|**62.5(+4.9)**|\n\n|DeepSeek-R1-Distill-Llama-8B|LogicBench|LogiQA2.0|FOLIO|AGIEval-LR|AGIEval-AR|ML-D1|ML-D2|ML-D3|ML-D4|ML-D5|Avg|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|Vanilla|83.1|53.4|53.1|53.3|30.4|78.0|67.1|60.0|49.2|46.6|57.4|\n|+LogicTree|91.2(+8.1)|54.9(+1.5)|57.8(4.7)|57.3(+4.0)|33.9(+3.5)|86.3(+7.3)|73.3(+6.2)|66.0(+6.0)|62.5(+13.3)|56.6(+10.0)|**63.9(+6.5)**|\n\nAs shown in the table, LogicTree consistently improves the performance of DeepSeek-R1-Distill-Llama-8B and DeepSeek-R1-Distill-Qwen-7B across multiple benchmarks, with average gains of 6.5% and 4.9%, respectively. Notably, on logic-intensive benchmarks such as the ML series, the improvements reach up to 13.3% and 10.9%. These results further confirm the effectiveness of LogicTree in enhancing the logical reasoning capabilities of LLMs.\n\nWe humbly hope our response has addressed your concerns. If you have any additional concerns or comments that we may have missed in our responses, we would be most grateful for any further feedback from you to help us further enhance our work."}, {"review_id": "Y3WtBWEkW0", "reviewer": "Reviewer_xJSd", "summary": "This paper introduces LogicTree, a framework for synthesizing multi-step logical reasoning datasets designed to enhance the complex reasoning capabilities of large language models (LLMs). The core idea involves generating symbolic logical reasoning trees via backward deduction using structural pattern matching, and then instantiating these trees into natural language using a two-stage LLM-based prompting strategy. This approach enables the construction of realistic, context-rich scenarios that support generalizable reasoning. The authors demonstrate that LogicTree outperforms several baselines across multiple logical reasoning benchmarks, particularly in multi-step reasoning tasks, achieving an average improvement of 10.5% in accuracy.", "questions": "Q1. The paper emphasizes the effectiveness of the method on smaller models, but then why are 1B or 3B-scale models not evaluated?\n\nQ2. Given that recent advances in long-chain Chain-of-Thought (CoT) prompting have significantly improved reasoning capabilities in language models, have you conducted any experiments or comparisons related to this line of work?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nQuality: The paper presents a well-designed methodology that combines symbolic logic with modern LLMs for dataset synthesis. The experimental results are comprehensive and show clear improvements over strong baselines.\n\nClarity: The writing is clear, with helpful diagrams and algorithm pseudocode.\n\nSignificance: LogicTree contributes meaningfully to enhancing logical reasoning abilities in LLMs, particularly in complex, multi-step settings.\n\nOriginality: While the use of tree structures for logical reasoning is not new and has been employed in prior works such as proof trees, symbolic reasoning chains, and MCTS-based methods, this paper distinguishes itself through its combination of structural pattern matching-based logic tree generation and a two-stage LLM-driven instantiation process. \n\nWeaknesses:\n\nQuality: While the paper provides strong empirical evidence of LogicTree’s effectiveness on multi-step reasoning, it lacks a formal analysis of algorithmic complexity (e.g., runtime or space complexity) for logic tree generation or instantiation. There is no statistical breakdown of the structural complexity of the generated logic trees (e.g., average depth, node count, rule variety), which would help quantify dataset diversity and difficulty. The paper includes Qwen2.5-7B results only in Multi-LogiEval, and omits it from the main result table (Table 1) and other benchmark evaluations, limiting the completeness of cross-model comparison.\n\nClarity: Some table captions do not totally match with the content or the context around. \n\nSignificance: While LogicTree achieves significant performance gains on smaller models (e.g., +10.5% for LLaMA3-8B), the improvement becomes marginal on stronger models like LLaMA3-70B (+2.3%). The paper devotes substantial space to analyzing the weakest model, but lacks a discussion on why stronger models benefit less. This raises concerns about the long-term utility and scalability of the method, especially as modern 8B-class models (e.g., Qwen3-8B, DeepSeek-VL) continue to improve.", "was_revised": true, "final_justification": "The rebuttal addressed my concerns. No issues remain unresolved. \nHowever, the authors supplemented a significant amount of experiments during the rebuttal phase—experiments that should have been part of the original submission. I am uncertain whether this is permissible.", "rebuttal": "**Thank you for your valuable and constructive review. We sincerely appreciate your feedback. Below, we respond to each of your comments in detail and hope that our responses could properly address your concerns. If you find them satisfactory, we would be grateful if you could consider updating your score. Otherwise, please feel free to let us know any remaining questions or concerns, and we will be happy to continue the discussion.**\n\n# Weakness 1\n### Weakness 1.1\n> it lacks a formal analysis of algorithmic complexity (e.g., runtime or space complexity) for logic tree generation or instantiation. \n\nWe appreciate your attention to the computational complexity of our data synthesis process, which is indeed crucial for the practical applicability of our method. Below, we provide a detailed analysis of the algorithmic complexity for both logic tree generation and LLM-based instantiation:\n\n* **Logic tree generation**: As illustrated in Algorithm 1, the process of generating a symbolic reasoning tree has a **time complexity of $O(NM)$**, where N is the number of reasoning steps and M is the number of available logical rules. **Since both N(<20) and M(<200) are relatively small in practice, the computational cost of this step is negligible**.\n* LLM-based instantiation: The computational cost of calling LLMs is typically measured by **the number of output tokens they generate**, which serves as a practical indicator of runtime and resource usage. Specifically: 1) The first call to LLM populates the abstract logical symbols with concrete natural language entities. We denote the number of output tokens consumed in this step as $c$. 2) The second call translates the sequence of reasoning steps into natural language. We assume that translating each individual reasoning step requires an average of $T$ output tokens. The computational cost of generating a data instance with reasoning steps $m$ is:\n  $$Cost_{inst}=m⋅T+c$$\nFor a more detailed analysis of the complexity of invoking LLMs, please refer to Appendix B.5.\n\nOur framework guarantees correctness by design, thereby eliminating the need for multiple LLM calls. As a result, the generation cost remains acceptable compared to other approaches.\n\n### Weakness 1.2\n>  There is no statistical breakdown of the structural complexity of the generated logic trees (e.g., average depth, node count, rule variety),\n\nWe sincerely thank you for your insightful comment and apologize for the lack of this information in the previous version. We provide the relevant details below.\nThe table below presents statistical information for **logic trees with 5 to 8 reasoning steps**, including (1) the average maximum depth of the trees, (2) the average number of nodes per tree, (3) the average number of distinct rules applied, as well as (4) the proportion of first-order logic rules and propositional logic rules used in each tree.\n\n\n\n|Step|Number of Nodes|Maximum Depth|Number of Distinct rules|Proportion of Fol|Proportion of Prop|\n|-|-|-|-|-|-|\n|5|$11.46_{\\pm1.19}$|$3.68_{\\pm1.42}$|$4.14_{\\pm0.52}$|$31.42\\%$|$68.58\\%$|\n|6|$13.46_{\\pm1.01}$|$5.04_{\\pm0.32}$|$4.75_{\\pm0.81}$|$30.45\\%$|$69.55\\%$|\n|7|$15.78_{\\pm1.81}$|$5.61_{\\pm0.56}$|$5.29_{\\pm0.72}$|$29.94\\%$|$70.06\\%$|\n|8|$18.30_{\\pm2.01}$|$6.04_{\\pm0.76}$|$5.67_{\\pm0.97}$| $28.82\\%$|$71.18\\%$|\n\n### Weakness 1.3\n> The paper includes Qwen2.5-7B results only in Multi-LogiEval, and omits it from the main result table (Table 1) and other benchmark evaluations, limiting the completeness of cross-model comparison.\n\nThank you for your valuable comments. We apologize for omitting this part of the evaluation due to space limitations in the main text. We have now added the experimental results of the Qwen2.5-7B model as follows.\n\n|Qwen2.5-7B|LogicBench|LogiQA2.0|FOLIO|BBH-Logical|AGIEval-LR|AGIEval-AR|Avg|\n|-|-|-|-|-|-|-|-|\n|Vanilla|83.1|63.8|54.3|72.8|65.3|23.9|60.4|\n|+PAPARULE|82.9|**65.1**|55.3|72.8|68.3|24.5|61.5|\n|+LogicAsker|84.4|64.9|58.7|72.8|66.7|24.5|62.1|\n|+FLD$_{\\times 2}$|83.4|**65.1**|59.6|73.6|67.3|26.1|62.5|\n|LogicTree|**89.2(+6.1)**|$\\underline{64.9}$(+1.1)|**63.8(+9.5)**|**74.3(+1.5)**|**69.3(+4.0)**|**28.7(+4.8)**|65.0|\n\nAs shown in the table, LogicTree demonstrates consistent and comprehensive performance improvements on Qwen2.5-7B, in line with our main experimental results. On average, it achieves a 4.6% increase in accuracy across multiple benchmarks. Notably, it achieves gains of 6.1% on LogicBench and 9.5% on FOLIO, both of which require more advanced logical reasoning capabilities.\n# Weakness 2 \n> Some table captions do not totally match with the content or the context around.\n\nThank you for your careful review and valuable comments. We will thoroughly check the formatting and content throughout the paper and make the necessary corrections in the final version.\n# Weakness 3\n> The paper devotes substantial space to analyzing the weakest model, but lacks a discussion on why stronger models benefit less...\n\nThank you for your valuable comments. First, we would like to clarify that the average improvement for LLaMA3.1-70B is relatively modest (2.3%). This is primarily because LLaMA3.1-70B already achieves high scores on several relatively simple benchmarks, leaving limited room for further improvement. It is worth noting that the LLaMA3.1-70B model still achieves substantial improvements on several challenging benchmarks, such as a **12.1%** gain on MultiLogic-Eval(Appendix Table 5) and a **5.6%** gain on FOLIO.\n\nTo further address your concerns, we conducted additional experiments using the well-known **DeepSeek series of distilled models**.\n\n|DeepSeek-R1-Distill-Qwen-7B|LogicBench|LogiQA2.0|FOLIO|AGIEval-LR|AGIEval-AR|ML-D1|ML-D2|ML-D3|ML-D4|ML-D5|Avg|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|Vanilla|86.8|52.4|61.2|56.8|32.6|70.0|57.0|60.0|55.0|44.4|57.6|\n|+LogicTree|90.6(+3.8)|53.4(+1.0)|63.4(2.2)|62.4(5.6)|34.4(+1.8)|80.9(+10.9)|67.6(+10.6)|62.5(+2.5)|56.6(+1.6)|53.3(+9.9)|**62.5(+4.9)**|\n\n|DeepSeek-R1-Distill-Llama-8B|LogicBench|LogiQA2.0|FOLIO|AGIEval-LR|AGIEval-AR|ML-D1|ML-D2|ML-D3|ML-D4|ML-D5|Avg|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|Vanilla|83.1|53.4|53.1|53.3|30.4|78.0|67.1|60.0|49.2|46.6|57.4|\n|+LogicTree|91.2(+8.1)|54.9(+1.5)|57.8(4.7)|57.3(+4.0)|33.9(+3.5)|86.3(+7.3)|73.3(+6.2)|66.0(+6.0)|62.5(+13.3)|56.6(+10.0)|**63.9(+6.5)**|\n\nAs shown in the table, LogicTree consistently improves the performance of DeepSeek-R1-Distill-Llama-8B and DeepSeek-R1-Distill-Qwen-7B across multiple benchmarks, with average gains of **6.5%** and **4.9%**, respectively.\n# Question 1\n> The paper emphasizes the effectiveness of the method on smaller models, but then why are 1B or 3B-scale models not evaluated?\n\nThank you for your valuable comments. We have conducted additional experiments using Qwen2.5-1.5B and Qwen2.5-3B to further evaluate the effectiveness of LogicTree on smaller language models.\n\n\n|Qwen2.5-1.5B|LogicBench|LogiQA2.0|FOLIO|AGIEval-LR|AGIEval-AR|ML-D1|ML-D2|ML-D3|ML-D4|ML-D5|Avg|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|Vanilla|65.0|43.9|48.5|39.5|17.4|65.6|53.3|48.8|43.3|31.1|45.8|\n|+LogicTree|74.4|48.4|53.4|42.4|21.3|80.9|56.2|54.8|54.2|44.4|**53.5(+7.7)**|\n\n|Qwen2.5-3B|LogicBench|LogiQA2.0|FOLIO|AGIEval-LR|AGIEval-AR|ML-D1|ML-D2|ML-D3|ML-D4|ML-D5|Avg|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|Vanilla|74.4|53.9|51.0|53.5|20.9|74.5|55.2|46.5|45.0|36.6|51.2|\n|+LogicTree|83.9|56.2|61.2|55.7|24.4|86.3|67.6|59.3|58.3|50.0|**60.5(+9.3)**|\n\nAs shown in the table, LogicTree effectively enhances the logical reasoning capabilities of smaller models such as Qwen2.5-1.5B and Qwen2.5-3B, achieving consistent and substantial improvements across multiple benchmarks. Specifically, Qwen2.5-1.5B achieves an average accuracy gain of 7.7%, and Qwen2.5-3B improves by 9.3%, demonstrating that our synthesized data remains effective even for smaller-scale models.\n# Question 2\n> Given that recent advances in long-chain Chain-of-Thought (CoT) prompting ..., have you conducted any experiments or comparisons related to this line of work?\n\nWe appreciate you pointing out this critical issue—it indeed relates to one of our core contributions. In fact, we provide **a detailed discussion of this aspect in Appendix A.1**. We apologize for not highlighting it more clearly in the main text, which may have caused confusion. To address this, we will add an appropriate reference in Section 5.\n\n**For your convenience, we summarize the key differences below**:\n\nMany efforts focus on designing more effective prompting strategies to help LLMs complete complex logical reasoning tasks, such as Symbolic Chain-of-Thought (SymbCoT)[1]  and Logic-of-Thought[2]. Although prompt-based methods can effectively leverage the potential of large language models(LLMs), their performance remains fundamentally constrained by the models’ inherent reasoning capabilities. \n\n**These works aim to elicit the existing reasoning capabilities of LLMs through carefully designed prompts, rather than enhancing their intrinsic reasoning abilities**. In contrast, our approach focuses on synthesizing a large volume of high-quality logical reasoning data from a training perspective, which helps to strengthen the inherent reasoning skills of LLMs. **Moreover, our method is not in conflict with these prompt-based approaches — rather, they are complementary. We believe that developing a unified framework that bridges training-based and prompt-based methods is a promising direction for future research**.\n\nWe humbly hope our response has addressed your concerns. If you have any additional concerns or comments that we may have missed in our responses, we would be most grateful for any further feedback from you to help us further enhance our work.\n\n## Reference:\n[1] Faithful logical reasoning via symbolic chain-of-thought. ACL, 2024.\n\n[2] Logic-of-thought: Injecting logic into contexts for full reasoning in large language models. NAACL,2024"}, {"review_id": "Uxp6Bq4NbM", "reviewer": "Reviewer_G8Cd", "summary": "This paper describes an approach to try and generate new synthetic data for logical inference (both propositional logic and first order logic are covered) with the goal of improving reasoning in LLMs.  The problem is an important one in the era of the so-called reasoning models.  The approach described here tries to define a 'LogicTree' which effectively describes a tree of rules applied using backward chaining, have an LLM assign realistic statements to each rule based on a specified domain, create a natural language reasoning process which presumably defines the steps in the reasoning process and then use that to fine tune a set of models.  Results are reported on a set of base models and it is shown that this benchmark training improves performance on other logic related datasets.", "questions": "Please see questions above.", "limitations": "This section was outside of the 9 page limit, following references and had little substance.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1.  The approach is interesting, as is the creation of natural language versions of logical rules.  \n2.  The use of other logical benchmarks to see if the results generalize is also useful.\n\nWeaknesses:\n1.  I do not understand the attempt to define the LogicTree using some sort of 'pattern matching' structure matching approach - surely instrumenting any backward chaining symbolic system is likely to give you the same sort of tree?  Why re-invent what is well known and well implemented?\n2.  Key details are lacking making the paper difficult to read, and hard to assess.  \nExamples: \na.  How were the initial problems specified?  Were they chosen from some existing logical benchmarks?\nb.  How many such problems were there, \nc.  What is the size/complexity of the problems - how many were propositional, and how many were first order logic.  \nd.  How did you assure that you had similar complexity problems to benchmarks from FOL theorem provers? \ne.  How do you assure that when problems are indeed complex enough (such as those problems used by FOL theorem provers) the NL statements make sense at a semantic level?\n3.  If the core contribution is the benchmark then the focus should be on benchmark characteristics and how they differ from other benchmarks.  If the core contribution is some sort of novel mechanism to teach LLMs reasoning the novelty was not clear.\n\nBased on the author rebuttal I clearly misunderstood the main contribution here.  Thank you for the explanation - I will edit my scores appropriately.", "was_revised": true, "final_justification": "I considered the authors' explanation of the core contribution and realized I had misunderstood it.", "rebuttal": "**Thank you for your valuable and constructive review. Below, we respond to each of your comments in detail and hope that our responses could properly address your concerns. If you find them satisfactory, we would be grateful if you could consider updating your score (Rating: 2: reject). Otherwise, please feel free to let us know any remaining questions or concerns, and we will be happy to continue the discussion.**\n\n# Motivation and Core Contributions of Our Work\nWe sincerely apologize for any lack of clarity in our manuscript that may have caused confusion. To provide a clearer understanding of our work, we would like to begin by outlining the motivation and core contributions of this paper.\n### 1. Motivation: The Scarcity of High-Quality, Multi-Step Logical Reasoning Data for LLM Training\n\nOur research is motivated by a critical challenge: **LLMs still exhibit significant weaknesses in complex, multi-step logical reasoning**. This limitaion underscore a significant scarcity of high-quality training data that embodies multi-step logical reasoning chains in natural language form. \n\nPrior work on synthesizing logical data suffers from key limitations: 1) **Limited Reasoning Complexity**, with limited combinations of logical rules.2) **Inadequate Real-World Instantiation**, lacking contextual coherence and fail to reflect realistic scenarios.\n\n### 2. Our Main Contribution: The LogicTree Framework for Synthesizing Logical Reasoning Datasets for LLM Training\nWe introduce LogicTree, **a novel framework for synthesizing multi-step logical reasoning datasets for LLM training**, offering advantages in terms of complexity and instantiation.  **The pipeline consists of two main stages**. First, we construct symbolic and diverse formal logic trees from scratch. Then, **we design a two-stage LLM-based method to instantiate these symbolic logic trees** into context-rich natural language narratives with diverse real-world scenarios.\n\nOur experiments show that **fine-tuning LLMs on the LogicTree-generated dataset** yields significant and consistent improvements across multiple challenging natural language logical reasoning benchmarks.\n\n# Weakness 1\n> ...any backward chaining symbolic system is likely to give you the same sort of tree? Why re-invent what is well known and well implemented?\n\nWe appreciate the reviewer‘s observation regarding a crucial point that we wish to clarify:  the fundamental difference in purpose between LogicTree and backward-chaining symbolic systems. **Our work is not an attempt to reinvent backward chaining for proof verification, but rather to design a novel framework for data synthesis to train LLMs**. \nTo address your question in detail, we organize our response along the following two aspects:\n## 1. Key Difference in Core Objective: Synthesis, Not Verification\n\n### 1.1 Backward chaining symbolic system for Verification\nOur understanding of these systems [1,2] are used for verification. Given a target conclusion or hypothesis, they **employ a backward-chaining mechanism to search a knowledge base for relevant facts and rules**. If a valid reasoning path is found, the system **outputs a grounded proof tree that supports the target conclusion**. For example: \n\n```\n        Conclusion: is(charlie, cold)?  \n                        |\nRule 1. IF is(X, young) And is(X, round), THEN is(charlie, cold).   \n                /                 \\\nFact 1: is(Bob, young)         Fact 2: is(bob, round).\n```\n\n\n**Backward-chaining symbolic systems are not well-suited for generating training data aimed at teaching LLMs logical reasoning**.\n1. The process of searching for a reasoning path is deterministic and does not allow control over the number of reasoning steps, tree structure, or types of rules used. \n2. The resulting proof tree is already instantiated — the validity of intermediate rules relies on specific factual knowledge. As a result, it lacks structural abstraction, making it unsuitable for synthesizing multiple diverse instantiations from the same underlying logic tree.\n### 1.2 LogicTree for Data Synthesis\nIn contrast, LogicTree is a synthesis framework that **starts from an arbitrary fully symbolic logical formula**. Guided by controllable specifications—such as the number of reasoning steps, rule types, and tree structure—it employs a backward-chaining strategy to **select applicable logical rules from a formal logic system**.\n```\n        Logic formula: P -> Q \n                | Rule: ((p → q)) ∧ (q → r)) ⊢ (p → r)\n             /      \\\n Premise1: P->T     Premise2: T->Q\n```\nBy instantiating abstract logic trees into diverse real-world scenarios, we aim to synthesize data that helps LLMs acquire fundamental logical reasoning skills.\n### 1.3 Key Differences\n| |Backward chaining symbolic system |LogicTree|\n|-|-|-|\n|Inputs|A specific conclusion within a knowledge base|A fully symbolic logical expression|\n|Outputs|An instantiated proof path based on facts from the knowledge base|A structured, abstract logical reasoning tree|\n|Rules |A predefined set of rules in the knowledge base| General logical rules applied flexibly via pattern matching|\n|Objective|Verification: To find an existing proof that supports the conclusion|Synthesis: To construct new, structurally controllable logic from scratch|\n\n## 2. Motivation for Structural Pattern Matching in Logical Formulas\nUnder the synthesis objective, existing methods[3,4,5] for generating formal logic trees face a key limitation: **they require strict structural identity between formulas**, which severely limits the diversity of rule combinations.\n\nTo overcome this, **we introduce Structural Pattern Matching—a more flexible backward deduction method that compares the structural skeletons of Abstract Syntax Trees** (ASTs, line 113). For example, consider the abstract rule of Hypothetical Syllogism, `HS: ((p→q)∧(q→r)) ⊢ (p→r)`. A traditional method requiring unification could only apply this to a simple `X→Y` conclusion and would fail with a more complex formula like `(A∨B)→(¬C)`. Our method enables us to apply the HS rule to decompose `(A∨B)→(¬C)` into two new sub-goals: `(A∨B)→G` and `G→(¬C)`. \n\n# Weakness 2\n\nThank you for your valuable comments. We will respond to your questions point by point. \n\n### Question 2.1\n\n > a. How were the initial problems specified? Were they chosen from some existing logical benchmarks? \n\nWe generate the entire logical reasoning tree from scratch based solely on fundamental logical rules, **without the need for any initial problems as input**. As discussed in more detail in Weakness 1.2 above, our reasoning tree generation starts from an arbitrary, fully symbolic logical expression — it is not derived from any real-world problem. \n\n\n### Question 2.2\n>b. How many such problems were there, \n>c. What is the size/complexity of the problems. \n\nWe generated 5k symbolic logic trees with **depths from 2 to 15**, and instantiated each into **3 semantically diverse scenarios**, yielding 1.5k reasoning problems. After applying an automatic verification process that discarded 8.73% of invalid samples, **the final dataset for LLM training contains **1.38k** high-quality, multi-step reasoning instances**.\n\n\nDue to space limitations, the table below presents statistical information for logic trees with 5 to 8 reasoning steps.\n| Step|Number of Nodes|Maximum Depth|Number of Distinct rules|Proportion of Fol|Proportion of Prop|\n|-|-|-|-|-|-|\n|5|$11.46_{\\pm1.19}$|$3.68_{\\pm1.42}$|$4.14_{\\pm0.52}$|$31.42\\%$|$68.58\\%$|\n|6|$13.46_{\\pm1.01}$|$5.04_{\\pm0.32}$|$4.75_{\\pm0.81}$|$30.45\\%$|$69.55\\%$|\n|7|$15.78_{\\pm1.81}$|$5.61_{\\pm0.56}$|$5.29_{\\pm0.72}$|$29.94\\%$|$70.06\\%$|\n|8|$18.30_{\\pm2.01}$|$6.04_{\\pm0.76}$|$5.67_{\\pm0.97}$|$28.82\\%$|$71.18\\%$|\n\n\n### Questino 2.3\n\n> d. How did you assure that you had similar complexity problems to benchmarks from FOL theorem provers?\n> How do you ... the NL statements make sense at a semantic level? \n\nSince LLMs still face significant challenges in multi-step logical reasoning [6] — for example, their performance on 4-step and 5-step problems remains weak in Multi-LogiEval — **we focus on synthesizing problems that are currently difficult for LLMs to solve, in order to better support their learning**.\n\nWe emphasize that our work is not intended as a benchmark for first-order logic provers, nor does it require alignment with their difficulty levels. Rather, **our objective is to generate logically valid, diverse, and controllable training samples that are suitable for improving LLM reasoning capabilities**.\n\nIn our synthesis pipeline, we design a two-stage LLM-based approach to instantiate logic trees into realistic and coherent scenarios. The goal is not to have LLMs memorize fixed natural language statements, but rather to help them learn how fundamental logical reasoning rules are applied across diverse contexts. Extensive evaluations on multiple benchmarks demonstrate the effectiveness of our method.\n\n\n# Weakness 3\n> If the core contribution is ...\n\n\nAs we stated at the beginning of our reply,  our work aims to address the scarcity of high-quality, multi-step logical reasoning data for LLM training. We propose LogicTree, a framework that synthesizes instantiated and structurally complex logical reasoning data. Such data is essential for teaching LLMs to develop robust, generalizable logical reasoning capabilities.\n\n\n\n### Reference：\n[1] Lambada: Backward chaining for automated reasoning in natural language. ACL, 2023. \n\n[2] SymBa: Symbolic Backward Chaining for Structured Natural Language Reasoning. ACL, 2025. \n\n[3] Enhancing reasoning capabilities of llms via principled synthetic logic corpus. NIPS, 2024. \n\n[4] Learning deductive reasoning from synthetic corpus based on formal logic. ICML, 2023. \n\n[5] LogicAsker: Evaluating and improving the logical reasoning ability of large language models. EMNLP, 2024. \n\n[6] Multi-logieval: Towards evaluating multi-step logical reasoning ability of large language models. EMNLP, 2024."}]

Author rebuttal:

Meta-review: The authors introduce “LogicTree”, a framework for synthesizing multi-step logical reasoning datasets with a focus on complexity and real-world contextual instantiation. The approach uses structural pattern matching for backward deduction to generate logic trees governed by first-order logic rules, and then employs a two-stage LLM-based method to instantiate these symbolic trees into diverse, context-rich, natural language scenarios. Logical consistency is checked post-hoc to filter poor samples. Experiments across several logical reasoning benchmarks demonstrate that training LLMs with LogicTree data improves accuracy over baselines.

Strengths of the paper:
- The logic tree generation based on structural pattern matching is well-grounded in formal logic.
- The experiments benchmark the method across diverse challenging datasets and tasks, including ablation studies for scenario instantiation and process diversity. Results support the main claims with quantitative improvements. 

The reviewers have initial concerns about computation cost, reliability of verification, dataset stats, and some technical details. The author's responses properly addressed all of these concerns. All reviewers are satisfied with the responses.
Decision: Accept (spotlight)