Title: LLM Meeting Decision Trees on Tabular Data
Venue: NeurIPS 2025
Accepted: yes
Keywords: Tabular data

Large language models

Classification and regression
Abstract: Tabular data have been playing a vital role in diverse real-world fields, including healthcare, finance, etc. 
With the recent success of Large Language Models (LLMs), early explorations of extending LLMs to the domain of tabular data have been developed. Most of these LLM-based methods typically first serialize tabular data into natural language descriptions, and then tune LLMs or directly infer on these serialized data. However, these methods suffer from two key inherent issues: (i) data perspective: existing data serialization methods lack universal applicability for structured tabular data, and may pose privacy risks through direct textual exposure, and (ii) model perspective: LLM fine-tuning methods struggle with tabular data, and in-context learning scalability is bottle-necked by input length constraints (suitable for few-shot learning). This work explores a novel direction of integrating LLMs into tabular data through logical decision tree rules as intermediaries, proposing a decision tree enhancer with LLM-derived rule for tabular prediction, DeLTa. The proposed DeLTa avoids tabular data serialization, and can be applied to full data learning setting without LLM fine-tuning. 
Specifically, we leverage the reasoning ability of LLMs to redesign an improved rule given a set of decision tree rules. Furthermore, we provide a calibration method for original decision trees via new generated rule by LLM, which approximates the error correction vector to steer the original decision tree predictions in the direction of ``errors'' reducing.
Finally, extensive experiments on diverse tabular benchmarks show that our method achieves state-of-the-art performance.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces DeLTa (Decision Tree Enhancer with LLM-derived Rule for Tabular Prediction), a novel approach that integrates Large Language Models (LLMs) with decision trees to improve predictions on tabular data. According to authors, DeLTa addresses three key issues in prior LLM-based methods for tabular data:\n1) lack of universal applicability and privacy risks from data serialization usually used in current LLM-based tabular approaches.\n2) challenges of LLM fine-tuning\n3) scalability limitations in-context learning\n\nThis approach seems to be using logical decision tree rules as intermediaries and it leverages the reasoning ability of LLMs to generate improved rules from existing decision tree rule sets without requiring LLM fine-tuning.\n\nSTRENGTHS AND WEAKNESSES:\nThe authors seem to deliver on their promise to avoid serialization, fine tuning and scalability. The main advantage of the paper seems to rely on improved rules from existing decision tree rule sets and using these rules to calibrate predictions. The method achieves state-of-the-art performance on diverse tabular benchmarks in both full-data and low-data regimes.\n\nVery interesting approach to combine 2 steps to refine tree rules and training a gradient net to correct errors.\n\nMy main comment is that this seems more of decision tree enhancer system that introduces advantages natural to LLMs rather than a LLM-based solution for tabular data problems. And that's fine, the way the authors use an LLM to synthesize a refined rule r* from multiple tree rules, and then apply that to structure a residual correction layer, reminds me of boosting. Would you say DeLTa is a boosting-inspired framework but the LLM refines instead of being a data-driven iterative fitting?\n \nMaybe also citing TabNet (it's one of the most well-known deep learning models for tabular data with attention and interpretability), or TabTransformer (pioneered using transformers on column embeddings).\n\nTypos:\n- throughough\n- an domain\n\nQUESTIONS:\n- The rule r* may contain threshold or other information depending on the feature types. Does this leak privacy in some form?\n- Apart from the experimental results, which are showing good performance, how can we measure if the LLM rewriting rules (or correcting them) is performing well? In a way, this is also a bit of a zero-shot task for the LLM itself.\n- How is η set? what's the interpretation of it in the context of DeLta?\n- Is it fair to say that the method always assumes that the LLM will always find a r* leading to lower loss but this is an assumption?\n\nLIMITATIONS:\nMaybe clarify:\n- Privacy leakage beyond samples\n- Discuss when LLM fails to improve rules", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper uses a LLM to generate new rules for decision trees. The paper proposes a novel gradient update strategy to incorporate LLM rules into the decision tree inference.

STRENGTHS AND WEAKNESSES:
Strengths:
- This paper has impressive performance compared to XGBoost, CatBoost, and TabPFN
- The method of using LLMs to augment decision trees is quite novel

Weaknesses:
- What is the intuition behind using LLMs to split v.s. classical methods, such as information gain, XGBoost, and CART. I understand the performance is better, but feel hesitant to accept the paper without understanding why the number is better. Are there specific examples where the LLM can generate a better rule than decision trees? Why do traditional techniques, like information gain, not capture such a rule?
- Using an LLM to generate rules sounds a lot more compute intensive than XGBoost or CART. What is the performance overhead of using the LLM?
- The datasets used in the paper is not very comparable. Why not use a standardized benchmark like WhyTrees or TabZilla? This makes the state-of-the-art argument a bit hard to accept.
- The writing could be improved. I had trouble reading through Section 4.2.

QUESTIONS:
See Weaknesses.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses key challenges in applying LLMs to tabular data, namely the limitations of data serialization and the difficulties of model fine-tuning. The proposed solution, DeLTa (Decision Tree Enhancer with LLM-derived Rule for Tabular Prediction), uses an LLM not to predict outcomes directly, but to refine a set of decision tree rules. First, a Random Forest model is trained on the data, and its decision rules are extracted. These rules are then fed into an LLM via a carefully constructed prompt, which instructs the LLM to synthesize a new, improved rule. This refined rule is then used to partition the data to train a \"Gradient Net\" that predicts and corrects the errors of the original Random Forest model. The authors demonstrate through extensive experiments that DeLTa achieves state-of-the-art performance on various classification and regression tasks, outperforming both LLM-based and conventional tabular data models. A key advantage of DeLTa is that it avoids direct data serialization, thus preserving privacy, and does not require fine-tuning the LLM, making it computationally efficient.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\nNovel Approach: The core idea of using an LLM to refine logical rules from decision trees, rather than serializing raw data, is highly innovative. It creates a new interface between LLMs and structured data models, moving beyond simple data-to-text conversion. \n\nWell-Written Framework: The paper presents the DeLTa framework in a clear, logical, and well-structured manner. The motivation is well-established, and the methodology is described with sufficient detail, making the novel concepts easy to follow. \n\nStrong Empirical Results: DeLTa shows promising and consistently strong performance across a variety of classification and regression datasets, under both full-data and few-shot settings. It outperforms other LLM-based methods and is competitive with traditional state-of-the-art models like XGBoost and CatBoost. \n\n\nPrivacy-Preserving and Efficient: By operating on decision rules instead of individual data samples, the method avoids exposing potentially sensitive data to the LLM. Furthermore, it does not require computationally expensive LLM fine-tuning, making it a more practical and scalable solution. \n\n\nWeaknesses\n\nLimited Qualitative Analysis: While the paper proves that the refined rules are better (e.g., lower intra-node distance), it doesn't provide enough insight into how they are better.  A qualitative analysis showing concrete examples of original vs. LLM-refined rules would be invaluable to understand the reasoning and \"world knowledge\" the LLM contributes.\n\nNeed for Broader Validation: Although the experiments are extensive, validation on an even wider range of datasets, particularly those with very high dimensionality or different types of feature distributions, would further solidify the claims of the model's generalizability.\n\nDependency on Powerful LLMs: The best results are achieved using GPT-4. The performance is likely sensitive to the quality of the LLM used, and the method's effectiveness might decrease with less capable models. The prompt engineering itself is also a critical, and potentially sensitive, component.\n\nQUESTIONS:\nCould you provide a concrete example of a set of initial decision rules from a dataset and the corresponding refined rule generated by the LLM? What specific logical simplifications, generalizations, or corrections does the LLM typically introduce?\n\nHow does the performance of DeLTa vary with the complexity of the base model? For instance, how does it change if the initial rules are derived from a much larger and deeper Random Forest versus a smaller one?\n\nThe paper states the method is \"domain knowledge-agnostic.\"  However, could the LLM be using its own embedded knowledge related to feature names (when not anonymized) to refine the rules? How was this potential confounding factor addressed in the experiments?\n\nHow sensitive is the model's performance to the structure and wording of the prompt used to guide the LLM's rule refinement? \n\nWhat is the failure mode of the LLM in this task? Are there instances where the LLM produces a rule that is syntactically incorrect or logically inferior to the original rules? If so, how are these cases handled?\n\nLIMITATIONS:\nThe paper itself acknowledges and discusses limitations in the appendix.  Based on the paper's content, the primary limitations are:\n\nThe performance of DeLTa is inherently tied to the reasoning capabilities of the underlying LLM. A less advanced LLM might not produce effective rule refinements. \n\nThe process relies on effective prompt engineering to guide the LLM. The quality of the refined rule could be sensitive to the specific prompts used. \n\nWhile the method is more efficient than fine-tuning, it still relies on querying a large, often proprietary, LLM, which can incur costs and latency. \n\nThere is a lack of deep qualitative insight into the \"reasoning\" the LLM applies. It is not fully clear what kinds of logical patterns or external knowledge the LLM uses to improve the rules.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an approach to improve decision trees with LLMs. Specifically, after fitting the CART decision tree, authors feed the decision rules from every tree into an LLM and prompt it to come up with a \"better\" rule. The LLM refined rule is then merged with CART via a gradient net.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n-The paper is well written and relatively easy to follow. The proposed approach is innovative and is novel, particularly the LLM rule refinement and merging via gradient net.\n-Results on real world datasets show improvements of the base CART approach as well as other tabular LLM adaptations.\n\nWeaknesses:\n-I think the proposed approach lacks rigor over what LLM is actually doing. Looking at the prompt in Figure 6 of the appendix, after CART rules, the LLM receives the following prompt: \"Based on the above information, please learn the rules evolving process and help me design a better rule like what cart used for inference to achieve higher performance. Please not just copy, please refine these rules and create a new better one.\" Without seeing any of the training data and how it is partitioned by the CART rules, what is a \"better rule\" here? Furthermore, looking at the example output in Figure 11 the LLM generates explanations like \"Feature `16` was chosen due to its significance in Tree 1 with a prominent threshold of `0.82`.\". Here again without seeing how training data is partitioned by this tree, \"significance\" and \"prominent threshold\" seem like random guesses. I think that a rigorous analysis of this step should be conducted to understand how an LLM can refine tree rules without seeing any data, before this paper can be accepted for publication.\n\nQUESTIONS:\nSee comments above regarding the LLM rule refinement and additional analysis of this step.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}