Title: LogicTree: Improving Complex Reasoning of LLMs via Instantiated Multi-step Synthetic Logical Data
Venue: NeurIPS 2025
Accepted: yes
Keywords: large language model

logical reasoning

data synthesis
Abstract: Despite their remarkable performance on various tasks, Large Language Models (LLMs) still struggle with logical reasoning, particularly in complex and multi-step reasoning processes. 
Among various efforts to enhance LLMs' reasoning capabilities, synthesizing large-scale, high-quality logical reasoning datasets has emerged as a promising direction. 
However, existing methods often rely on predefined templates for logical reasoning data generation, limiting their adaptability to real-world scenarios. 
To address the limitation, we propose **LogicTree**, a novel framework for efficiently synthesizing multi-step logical reasoning dataset that excels in both complexity and instantiation.
By iteratively searching for applicable logic rules based on structural pattern matching to perform backward deduction, **LogicTree** constructs multi-step logic trees that capture complex reasoning patterns. 
Furthermore, we employ a two-stage LLM-based approach to instantiate various real-world scenarios for each logic tree, generating consistent real-world reasoning processes that carry contextual significance.   This helps LLMs develop generalizable logical reasoning abilities across diverse scenarios rather than merely memorizing templates.
Experiments on multiple benchmarks demonstrate that our approach achieves an average improvement of 9.4\% in accuracy on complex logical reasoning tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper describes an approach to try and generate new synthetic data for logical inference (both propositional logic and first order logic are covered) with the goal of improving reasoning in LLMs.  The problem is an important one in the era of the so-called reasoning models.  The approach described here tries to define a 'LogicTree' which effectively describes a tree of rules applied using backward chaining, have an LLM assign realistic statements to each rule based on a specified domain, create a natural language reasoning process which presumably defines the steps in the reasoning process and then use that to fine tune a set of models.  Results are reported on a set of base models and it is shown that this benchmark training improves performance on other logic related datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1.  The approach is interesting, as is the creation of natural language versions of logical rules.  \n2.  The use of other logical benchmarks to see if the results generalize is also useful.\n\nWeaknesses:\n1.  I do not understand the attempt to define the LogicTree using some sort of 'pattern matching' structure matching approach - surely instrumenting any backward chaining symbolic system is likely to give you the same sort of tree?  Why re-invent what is well known and well implemented?\n2.  Key details are lacking making the paper difficult to read, and hard to assess.  \nExamples: \na.  How were the initial problems specified?  Were they chosen from some existing logical benchmarks?\nb.  How many such problems were there, \nc.  What is the size/complexity of the problems - how many were propositional, and how many were first order logic.  \nd.  How did you assure that you had similar complexity problems to benchmarks from FOL theorem provers? \ne.  How do you assure that when problems are indeed complex enough (such as those problems used by FOL theorem provers) the NL statements make sense at a semantic level?\n3.  If the core contribution is the benchmark then the focus should be on benchmark characteristics and how they differ from other benchmarks.  If the core contribution is some sort of novel mechanism to teach LLMs reasoning the novelty was not clear.\n\nBased on the author rebuttal I clearly misunderstood the main contribution here.  Thank you for the explanation - I will edit my scores appropriately.\n\nQUESTIONS:\nPlease see questions above.\n\nLIMITATIONS:\nThis section was outside of the 9 page limit, following references and had little substance.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces LogicTree, a framework for synthesizing multi-step logical reasoning datasets designed to enhance the complex reasoning capabilities of large language models (LLMs). The core idea involves generating symbolic logical reasoning trees via backward deduction using structural pattern matching, and then instantiating these trees into natural language using a two-stage LLM-based prompting strategy. This approach enables the construction of realistic, context-rich scenarios that support generalizable reasoning. The authors demonstrate that LogicTree outperforms several baselines across multiple logical reasoning benchmarks, particularly in multi-step reasoning tasks, achieving an average improvement of 10.5% in accuracy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nQuality: The paper presents a well-designed methodology that combines symbolic logic with modern LLMs for dataset synthesis. The experimental results are comprehensive and show clear improvements over strong baselines.\n\nClarity: The writing is clear, with helpful diagrams and algorithm pseudocode.\n\nSignificance: LogicTree contributes meaningfully to enhancing logical reasoning abilities in LLMs, particularly in complex, multi-step settings.\n\nOriginality: While the use of tree structures for logical reasoning is not new and has been employed in prior works such as proof trees, symbolic reasoning chains, and MCTS-based methods, this paper distinguishes itself through its combination of structural pattern matching-based logic tree generation and a two-stage LLM-driven instantiation process. \n\nWeaknesses:\n\nQuality: While the paper provides strong empirical evidence of LogicTree’s effectiveness on multi-step reasoning, it lacks a formal analysis of algorithmic complexity (e.g., runtime or space complexity) for logic tree generation or instantiation. There is no statistical breakdown of the structural complexity of the generated logic trees (e.g., average depth, node count, rule variety), which would help quantify dataset diversity and difficulty. The paper includes Qwen2.5-7B results only in Multi-LogiEval, and omits it from the main result table (Table 1) and other benchmark evaluations, limiting the completeness of cross-model comparison.\n\nClarity: Some table captions do not totally match with the content or the context around. \n\nSignificance: While LogicTree achieves significant performance gains on smaller models (e.g., +10.5% for LLaMA3-8B), the improvement becomes marginal on stronger models like LLaMA3-70B (+2.3%). The paper devotes substantial space to analyzing the weakest model, but lacks a discussion on why stronger models benefit less. This raises concerns about the long-term utility and scalability of the method, especially as modern 8B-class models (e.g., Qwen3-8B, DeepSeek-VL) continue to improve.\n\nQUESTIONS:\nQ1. The paper emphasizes the effectiveness of the method on smaller models, but then why are 1B or 3B-scale models not evaluated?\n\nQ2. Given that recent advances in long-chain Chain-of-Thought (CoT) prompting have significantly improved reasoning capabilities in language models, have you conducted any experiments or comparisons related to this line of work?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel framework called LogicTree for synthesizing multi-step logical reasoning datasets. The method leverages first-order logic rules to generate complex reasoning trees and instantiates them using Large Language Models to produce natural language reasoning data with realistic scenarios. The paper conducts experiments across multiple benchmarks and demonstrates a good improvement of accuracy on complex logical reasoning tasks.\n\nSTRENGTHS AND WEAKNESSES:\n## Paper Strengths:\n1. The proposed LogicTree framework is innovative in efficiently synthesizing multi-step logical reasoning datasets with complex reasoning patterns.\n2. The two-stage LLM-based instantiation technique for translating symbolic logical expressions into concrete statements with contextual significance is insightful. This approach enhances the realism and applicability of the generated datasets.\n3. The paper provides in-depth experimental results demonstrating the effectiveness of LogicTree in enhancing LLMs' reasoning abilities across various benchmarks and questions of different complexity levels.\n\n## Paper Weaknesses:\n1. Lack of details statistics on the generated datasets, such as the number of questions, unique reasoning patterns, complexity levels, and distribution of question types. This information is crucial for understanding the diversity and coverage of the synthesized datasets.\n2. The LLM-based instantiation purely relies on the LLMs' capabilities to generate natural language statements and inject factual knowledge. However, the LLMs suffer from hallucination issues, which may lead to the generation of incorrect or misleading statements with unreliable factual knowledge. The paper should provide more analysis on how the instantiation process handles such issues and ensures the correctness of the generated statements.\n3. The used baseline LLMs are just the vanilla LLMs without being specifically designed for intricate reasoning abilities. The paper should consider using more advanced large reasoning models, such as DeepSeek-R1-distill-8B, to further validate the effectiveness of the LogicTree framework in improving the reasoning abilities of LLMs.\n\nQUESTIONS:\n1. Can you provide more detailed statistics on the generated datasets?\n2. How does the LLM-based instantiation process handle the hallucination issues of LLMs to ensure the correctness of the generated statements? \n3. Can you consider using more advanced large reasoning models as DeepSeek-R1-distill-8B, to further validate the effectiveness of the LogicTree framework in improving the reasoning abilities of LLMs?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce “LogicTree,” a framework for synthesizing multi-step logical reasoning datasets with a focus on complexity and real-world contextual instantiation. The approach uses structural pattern matching for backward deduction to generate logic trees governed by first-order logic rules, and then employs a two-stage LLM-based method to instantiate these symbolic trees into diverse, context-rich, natural language scenarios. Logical consistency is checked post-hoc to filter poor samples. Experiments across several logical reasoning benchmarks demonstrate that training LLMs with LogicTree data improves accuracy over baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n* Methodological soundness: The paper clearly articulates the LogicTree pipeline, which systematically combines symbolic backward deduction with LLM-based scenario instantiation. The logic tree generation based on structural pattern matching is well-grounded in formal logic and illustrated stepwise.\n* Comprehensive evaluation: The experiments benchmark the method across diverse challenging datasets and tasks, including ablation studies for scenario instantiation and process diversity. Results support the main claims with quantitative improvements.\n\n**Weaknesses**\n* LLM-based verification: The logic consistency verification relies on LLMs themselves to make sure consistency between instantiated text and logic formulas (Section 3.3). This may introduces a potential failure mode where systematic instantiation errors might go undetected.\n* Depth in real-world evaluation: While the benchmarks are diverse and the accuracy improvements are robust, there is insufficient qualitative analysis to illustrate how the reasoning behavior of LLMs undergoes qualitative changes after training with LogicTree. For instance, do the models reason more explicitly, avoid hallucinations, or perform better in out-of-domain real-world reasoning? Specifically, on which types of questions are the improvements most significant, and on which do they even decline? More interpretive or failure-case analyses would strengthen the argument that such improvements stem from generalization rather than mere memorization.\n\nQUESTIONS:\n* What are the practical, computational costs of large-scale synthetic data generation and verification using LLMs, and do these limit applicability for broader academic use?\n* How sensitive are the results to the number of scenario instantiations per logic tree? Is there an upper bound after which gains saturate or reverse?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}