LLM Meeting Decision Trees on Tabular Data
Authors: Hangting Ye
Jinmeng Li
He Zhao
Dandan Guo
Yi Chang
Venue: NeurIPS 2025 spotlight ()
Keywords: Tabular data
Large language models
Classification and regression
Primary area: general_machine_learning
TL;DR: We propose to integrate LLMs into tabular data without data serialization and LLMs fine-tuning.

Abstract: Tabular data have been playing a vital role in diverse real-world fields, including healthcare, finance, etc. 
With the recent success of Large Language Models (LLMs), early explorations of extending LLMs to the domain of tabular data have been developed. Most of these LLM-based methods typically first serialize tabular data into natural language descriptions, and then tune LLMs or directly infer on these serialized data. However, these methods suffer from two key inherent issues: (i) data perspective: existing data serialization methods lack universal applicability for structured tabular data, and may pose privacy risks through direct textual exposure, and (ii) model perspective: LLM fine-tuning methods struggle with tabular data, and in-context learning scalability is bottle-necked by input length constraints (suitable for few-shot learning). This work explores a novel direction of integrating LLMs into tabular data through logical decision tree rules as intermediaries, proposing a decision tree enhancer with LLM-derived rule for tabular prediction, DeLTa. The proposed DeLTa avoids tabular data serialization, and can be applied to full data learning setting without LLM fine-tuning. 
Specifically, we leverage the reasoning ability of LLMs to redesign an improved rule given a set of decision tree rules. Furthermore, we provide a calibration method for original decision trees via new generated rule by LLM, which approximates the error correction vector to steer the original decision tree predictions in the direction of ``errors'' reducing.
Finally, extensive experiments on diverse tabular benchmarks show that our method achieves state-of-the-art performance.

LLM Meeting Decision Trees on Tabular Data
Hangting Ye', Jinmeng Li', He Zhao??, Dandan Guol ", Yi Chang 4.5w School of Artificial Intelligence, Jilin University! CSIRO's Data61 P; Monash University?; International Center of Future Science, Jilin University Engineering Research Center of Knowledge-Driven Human- Intelligence, MOE, China" {yeht2118, jimm9221] (a0a9922@@aaii /8mails jilu edu cn, he zhao@data61, csiro.ar {guodandan, , yi chang edu.
Abstract
Tabular data have been playing a vital role in diverse real-world fields, including healthcare, finance, etc. With the recent success of Large Language Models (LLMs), early explorations of extending LLMs to the domain of tabular data have been developed. Most of these I.M--baset methods typically first serialize tabular data into natural language descriptions, and then tune LLMs or directly infer on these scrialized data. However, these methods suffer from two key inherent issues: (i) data perspective: existing data serialization methods lack universal applicability for structured tabular data, and may pose privacy risks through direct textual exposure, and (ii) model perspective: LLM fine-tum fine-tuming methods struggle with tabular data, and by input length constraints (suitable for few-shot learning). This work explores novel direction of integrating LLMs into tabular data through logical decision tree rules as intermediaries, proposing a decision tree enhancer with L..M-derived rule for tabular prediction, Del.a. The proposed DeL.a avoids tabular data serialization, and can be applied to full data learning setting without LLM fine-tuning, Specifically, reasoning ability LLMs to redesign an improved rule given a set decision tree rules. Furthermore, we provide a calibration method for original decision trees via new generated rule by LLM, which approximates the error correction vector steer the original decision tree predictions in the direction of "errors" reducing. Finally, extensive experiments on diverse tabular benchmarks show that method achieves state-of-ther art performance. The source code is available at hhtpps:/// htap::///gtthin /ttpttttttuuu/...//// com com/HanggtingYe/DelTT.
in mn-oonexx learning scalability
leverage the
Introduction
Tabular data, typically organized in structured table format within relational database with rows and columns standing for the data samples and heterogencous features (e.g., categorical and numerical features), is fundamental in various real-world fields, including healthcare M.. advertising 22
finance (3]. etc. Given the heterogeneity of features [4] 6 71, decision tree- rree-based methods g 10
were found to be particularly suitable for tabular data [ [3]. While deep learning has led to breakthroughs in computer vision [14] and natural language processing [15], decision tree-based methods still outperform the majority of existing deep tabular methods tabular prediction tasks such as classification and regression. This superiority of decision tree- based methods is explained by the feature heterogeneity for tabular data, where neural networks struggle to learn the irregular target functions compared to decision tree-based methods [6].
Recently, LLMs have exhibited remarkable capabilities in natural language understanding and reasoning 17 18] sparking growing interest applying LLMs to structured tabular data tasks [19
"Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Table 1: Most existing LLM-based tabular prediction methods require access tabular samples and either train the LLMs or perform inference via in-context learning. Many of them are restricted to constrained settings such as few-shot or classification-only settings. The applicability of each scenario is determined on the experimental setups in the original papers. For instance, if method primarily focuses on for few-shot learning, we mark the few-shot setting as applicable (V); it can also be extended to full- data training, we additionally mark the full-data setting applicable (n).
Methods
accoss sample required
No LLM
No pre-traiing runing
LLM required
Few
Applicd seenazio
Clussifiation
Regression
TabLLM (2023)
20
LIFT (2023)
TE-BERT (2024)
X
GT (2024)
b3
X
x
SERSAL (2025)
(Unupervisal)
(Binary classificationn
(2024)
28
X
x
F (2004)
[4]
Del.a (Ours)
20 21 23 24 25
Typically, most of the existing methods first serialize tabular samples into natural language descriptions, and then tune LLMs or directly infer via in-context learning on these serialized data. Despite the success of LLMs on text-based tasks, leveraging LLMs to empower tabular predictions remains a challenging task. Through an in-depth investigation of prior LLM-basee approaches for tabular data, we identify two inherent characteristics hindering prediction:
(i) Data perspective: To bridge the modality gap between unstructured text and structured tabular data with heterogeneous features, most existing methods serialize tabular samples into text formats. For instance, TabLLM (20] converts sample into listed feature descriptions such as "The [column name] is [feature value]", I" and demonstrates that this template facilitates very-few-shor classification by leveraging the semantic priors of column names and values already encoded in LLMs. However, many real-world tabular datasets anonymize feature names using placeholder symbols for privacy (22) (e.g.. finance), reducing the effectiveness of serialization methods that rely on semantic feature descriptors. In addition, LLMs tend to be less sensitive to numerical features [26 21. The predefined templates with inserted values often yield text formats that make LLMs struggle to understand the inherent interactions among different features. Beyond this, directly exposing the serialized tabular samples containing raw feature values LLMs raises significant privacy concern,, especially in sensitive domains such as healthcare and finance, where data security is crucial 27 28]. These challenges make it difficult to design a universally suitable serialization template for tabular data.
One straightforward way to adapt LLMs to tabular prediction tasks is fine- tuning LLMs' parameters. For example, LIFT [9] investigated the fine-tuned GPT-3 models [1 on tabular data, revealing that the performance of fine-tuned L.MMs was roughly on par with traditional tabular prediction methods. Despite the effectiveness, fine-tuning LLMs remains challenging task even with recent parameter-efficiee fine-tuning methods (29]. especially in structured tabular data [19]. Another research line employs in-context learning by adding few-shot example demonstra- tions to the prompts without training, which is evidenced by LLMs' impressive capacity to learn from demonstrations included as part of the prompt [7 But most of these methods are inherently limited by the input length constraints of LLMs, restricting their use to few-shot, or classification only tasks. Therefore, LLMs still struggle to make satisfactory predictions for tabular data. We summarize these related works in Table
(ii) Model perspective:
In this paper, we explore novel direction of integrating LLMS into tabular data via logical decision tree rules (30) as intermediaries. We propose Del.Ta, a decision tree enhancer with LLM rule for tabular prediction. Specifically, we first leverage the reasoning ability of LLMs to redesign an improved rule given a decision tree rule set, enforcing greater coherence among samples falling into the same leaf node. The newly generated rule is used to approximate a sample-specific error correction vector to calibrate the prediction of original decision trees in the direction of "errors" reducing. The proposed DeLTa could well solve the aforementioned challenges: Solving the data issue: Unlike serialization methods that convert each sample into unnatural text formats, decision tree rules are composed of simple comparisons between feature values and thresholds, forming logical, interpretable structures that can be naturally expressed in text without relying on semantic columns names. In addition, decision tree rules represent global feature space partitioning rule rather than individual samples, which helps mitigate privacy concerns by avoiding exposure of sample-level

---

information. Solving the model issue: Moreover, the powerful reasoning ability of LLMs can be leveraged to redesign decision tree rules and help trees with aggregating their decisions, rather than directly using LLMs to generate label predictions. Notably, Del.Ta avoids serialize tabular data into natural language format, and does not require additional domain- comainn-ppecffic expertise or semantic information, such as explicit feature names and detailed task background knowledge. Furthermore, DeLTa can be applied in full data learning setting without LLM fine-tuning.
The contributions of this paper include: 1) We investigated the prior LLM-based methods for tabular prediction and explore a novel direction of integrating LLMs into tabular data via refining the decision tree rules, without directly accessing to the data itself, addressing the inherent issue w.r.t. data perspective and model perspective. 2) We propose a decision tree enhancer with LLM-derived rule for tabular prediction, DeLTa, which utilizes LLMs to refine set of decision tree rules derived on decision trees trained on multiple train subsets, where the newly generated rule used to infer the sample-wise error correction vector to calibrate the output of original decision trees. 3) We conduct extensive experiments on various tabular benchmarks and competing benchmark algorithms, and comprehensive results along with analysis and visualizations demonstrate our effectivvness.
2
Related work
Machine learning for tabular prediction. The development of effective algorithms for predictive modeling tabular data has been longstanding research topic. In the early days, decision tree- based methods (c.g. XGBoost [8]. CatBoost [TO] were found to be particularly suitable for tabular data. More recently, inspired by the success of of deep learning in computer vision (CV) [4] and natural language processing (NLP) [5], numerous methods have been proposed for tabular data to accomplish tabular prediction tasks. These works mainly include MLP-like models [3
5 32 33
attention- based architectures [34] [|3 (7), and retrieval-augmentn
architectures [38
35 39
Among these works, DCN V2
33 is an architecture that consists of an MLP-like module and
feature crossing module: AutoInt 34 leveraged the Transformer architecture to capture inter-column
correlations; FT-Transformer 5 further enhanced AutoInt's performance through improved token
embeddings; ModemNCA 38 makes predictions based on the relationships with neighbors in
learned embedding space. Recently, another line of research has tried to use additional information outside target dataset to enhance tabular data prediction. XTab 4] pretrains Transformer on variety of datasets for cross-table pretraining. TabPFN 42 43 which is pretrained on a large set of synthetic datasets, serves as foundation model for small to medium sized tabular data.
Large language models for tabular prediction. Motivated by the impressive success of LLMs [17 18 44 45). another promising study has attempted to harness the rich prior knowledge encapsulated by LLMS in tabular prediction tasks. LIFT [19] investigated the performance of fine-tuned GPT-3 models 17 on tabular data, revealing that the performance of fine-tuned LLMs was roughly on par with traditional solutions. Extending this line of research, TabLLM [2] employed TO (46] as the base LLM and demonstrated competitive performance of fine-tuned LLMs in very few-shot scenarios. Additionally, recent efforts have focused on pre-training LLMs over diverse tabular datasets from different domains. Among these, TP-BERTa [21] introduces a tabular-specific tokenization scheme, enabling single pre-trained language model to generalize across multiple tabular datasets after further fine-tuning GTL 22 further promotes comprehensive instruction
capabilities for both shot and in-context learning with a limited number of examples. Despite their promising results, these approaches typically require training or fine-tuning LLMs. Instead of directly fine- tuning LLMs, P2T (23 leverages unlabeled data correlated with target data expressed in natural language form and prompts L.LMs for few-shot semi-supervised learning. Recently, Summary [47 proposed a boosting framework that treats LLMs as weak learners for tabular prediction, particularly for tasks involving small numbers of data points. FeatLLM (24) prompts LLMs with serialized training examples to generate new features for few-shot classification tasks. SERSAL (25] explores to use synergy learning with FT-Transformer to enhance the noisy annotations generated by LLMs for binary classification tasks in unsupervised manner. This method also needs to fine-tune the LLMs. Going beyond tabular prediction, Nam et al. (48) leverages LLMs optimize the feature generator with tree rules in the field of feature engineering [49]. which we do not consider as a close related work to ours the primary goal, motivation, and methodology are different.
Although prior works demonstrate that L.LMs can make predictions for tabular tasks, the majority of these approaches rely on first converting tabular data into natural language descriptions, i.c.,
3

---

serialization, which often produces unnatural texts that differ from how humans might describe the data [47]. And it also challenging design such suitable template. In addition, these methods suffer from two limitations that either need to train LLMs or are restricted to constrained settings such as few-shot or classification-onl settings. Table summarizes these related works. The above limitations raise fundamental question: can we harness the capabilities of LLMs without fine- tuning and without serializing tabular data, to improve full-data tabular prediction tasks across both classification and regression?
3 Preliminaries
Problem formulation. Denote a tabular dataset = [Zx, yr as a collection of N samples, where (2i, (c) is the i-th data pair with z; E X representing input features and yi Y the corre- sponding label. Concretely, we have 2, = (z. (mm)) 2 a(cat)) where x; ((num) and (caat) represent the numerical and categorical features respectively. We consider supervised tabular prediction tasks: binary classification Y= {0, 13, multiclass classification Y= (1,.... c} and regression == For data splits, Dtrain denotes training set for model training, Deat validation set for early stopping and hyperparameter tuning, and Diest test set for final evaluation. The goal is to obtain an accurate model G: X Y trained on Drain that minimizes the expected loss (G y)]. Here, LL is smooth loss function (e.g., mean squared error or cross-entropy) and G can be any tabular predictive model.
Decision trees (DTs). A decision tree (DT) 300 is a hierarchical model that recursively partitions the input feature space into disjoint regions, i.c., Ieaf nodes, through a series of axis-aligned splits. For a given partitioning rule r, the prediction function f can be formally expressed as:
Node(r)
f(x| r)
Σ A AI- Li(r)),
(I)
where Node(r) denotes the number of leaf nodes in the tree, [((r) denotes the 1-th leaf node containing a set of x from Derain. T(.) is an indicator function determining if a of interest would be assigned to leaf node (r) based on rule r. and À X + Y represents leaf-specific prediction values. Typically, A will be the average of corresponding labels for the leaf node in regression tasks, or the empirical class distribution in classification tasks. The structure of the tree governed by a feature index vector & and a threshold vector T, which together define the decision path containing a series of internal nodes from the root to each leaf node. Specifically, decision tree constructed by recursively applying binary tests of the form z I Ty at internal nodes, where j indexes internal node along the root-to-lcaf path, l, specifies feature index for j-th internal node split, z denotes the corresponding feature value, and Ty defines corresponding splitting threshold. this work, we adopt CART (30] as our decision tree implementation due to two primary reasons: (i) it offers high interpretability and can be expressed using a simple if-else syntax, (ii) tree-based models, often ensembles of simple decision trees like CART, outperform deep learning approaches in numerous tabular prediction tasks 5
4 Proposed method: DeLTa
We propose a novel approach, Decision Tree Enhancer with LLM-derived Rule for Tabular Prediction (DeLTa), understanding the underlying rule logic encoded by decision trees and generating improved rules for tabular prediction via LLM, without requiring any fine-tuning of the LLM. This strategy introduces a fundamentally distinct interface for extending LLM capabilities to tabular tasks, departing from the majority of conventional data-to-text serialization pipelines that enforce LLMs to understand the serialized tabular data. In the following, refinement in Section 4.T then, we will provide the calibration method for original decision trees via new generated rule by LLM in Section 4.2 also give the overall implementation of ours in An overview of the proposed framework is depicted in Fig.
will elaborate on LLM-based decision tree rules
Section 4.3
4.1 I.LM-based decision tree rules refinement
The decision tree rules refinement procedure consists of the following stages: (i) decision tree rules initialization, that constructs a diverse decision tree rule set, and (ii) rule understanding via LLM,, that leverages the LLM to understand and refine the original rule set to obtain a new rulc.

---

NLA Bejoelooe F F
[+]
COPtonae DELLaGO Thee Pulse Belicmmnsl
Sugeott
Expert
PMde 2 100
Dale Set
128
holor"
[4)
Parlaced Raie-Gasded Decinaco Free Zrase Conrection
Icace
RIGGE cadacseg
Seftined baler'
-9
Figure 1:
DeLTa framework. shown in the main objective, we calibrate output of original decision tree experts F(x) in the direction of "errors" reducing. Subfig (a) describes the process of refining decision tree rules with LLM, and subfig (b) details the refined rule-guided error correction for decision trees.
Decision tree rules initialization. To mitigate the risk of overfitting for single decision tree expert [50], the classical Random Forest [51] algorithm trains diverse decision tree experts (fr }K on different subsets of 0 Dranns and uses the ensemble of {fk]R=] as prediction:
F(x)
K Σ fa(x//Dtrain Tk))
where F(x) denotes the label prediction produced by the Random Forest, K is the number of experts, D Drain denotes the subset, J'k denotes the rule for expert f& derived from Eq. I and R {rx) denotes a source decision tree rule set. In terms of decision tree rule, each rule represents a logical comparison between features and thresholds displayed in semantically rich syntax, which thus be used to partition the feature space into disjoint regions (i.e., leaf nodes) with high interpretability. A well understanding of these rules could generate better feature space partitioning method, thereby promoting statistical coherence among samples assigned to the same leaf node. Although Random Forest ensembles the outputs from all rules, the inherent relationships and interactions among rules in R are ignored. With the development of K, analyzing these independent rules is gradually becoming more and more difficult, let alone utilizing these rules partition feature space. To this end, propose to leverage L.LSs to analyze and summarize the rule set R into a refined rule due to the powerful logical reasoning ability of LLM.
Rule understanding via LLM. Our objective is to use an LLM to generate a new rule r* given construct prompt for the LLM that guides it to understand the diverse decision tree rule set R and synthesize the redesigned rule from the aggregated diverse knowl- edge. To effectively leverage the reasoning ability of LLM, the prompt p (please see Appendix [A.4) A.4) is designed to include: meta information Pmeta that describes the task objective, the extracted deciston tree rule description Prute that contains R, and the requirement Proquirement for rule refinement. The formulation of generating r* is given by:
original rule R. Specifically,
(LM LLM((mett + Prule Prequirement)
where is the concatenation of each prompt, and this process is achieved through querying the LLM,, without requiring any fine-tuning. Notably, our approach does not require additional domain- specific expertise or semantic information, such as explicit feature names or detailed task background knowledge. This makes it applicable to scenarios where such information is unavailable or incomplete. Instead of relying the intrinsic domain knowledge of tabular tasks, such as those in healthcare or finance, we frame the prob- lem purely
6.0
CART Rule
domain knowledge-agnostie machine learning task. Our method leverages LLMs to reason decision tree rule set R into rule r", rather than directly inferring from serial- ized tabular data. To verify that LLM could generate a better rule, where samples grouped within the same leaf node exhibit greater statistical similarity, we compute the intra-node sample distance over all leaf nodes partitioned by r*, and observe that the distance of r* is lower than original rule r E R, illus- trated in Fig. 2 Next, we will elaborate how leverage the new rule r" generated by LLM to enhance tabular predictions.
5.5
Refined Rule
5.0
3.5
3.0
BA
Car Different Dalasets
CR
Figure 2: Average intra-node dis- tance comparison.
5

---

4.2 Refined rule-guided decision tree error correction
The refined rule r" is summarized over the rule set R, and thus provides a potentially better strategy that partitions the feature space of Draain into disjoint leaf nodes (L)(r))) Node(r") with each leaf node containing subset of x from Drrain and Node(r") denoting the number of leaf nodes. To use r" for making predictions, the simplest strategies include: (1) treating it as a standalone decision tree by assigning the input decision trees F(z). However, the former may be too isolated, as it does not leverage the predictive power of the existing ensemble, while the latter may be too diluted, as it does not fully utilize the refined guidance provided by LLMs. This work introduces a novel way to leverage the power of r* together with F(z) inspired by the intuition that estimating the difference between predictions and labels, i.e., the residual errors is often easier than directly predicting the ground-truth labels [52 14] Intuitively, we first use r" to partition data samples into leaf nodes, expecting the samples in the same node are similar to each other. Motivated by the famous Gradient Boosting algorithm [53] 54]
specific leaf node, or (2) appending it the existing ensemble of
for the samples in each node, we learn a specific mapping function that predicts residual errors of F(x) for the samples in that node. These learned functions are subsequently used to adjust the predictions made by F(z), leading to improved accuracy. Since r' is designed to group similar samples effectively, the residuals within each leaf node are expected to be more structured and thus easier to predict.
Formally, we introduce the definition of the direction of prediction "errors" increasing and reducing. and the definition of a new set derived from Derain used for fitting the mapping functions as follows.
Definition 1. Prediction "errors" increasing and reducing. For a given training sample x, we approximate the "error" using the gradient V (( y). where o' is the loss function discussed in Section 3 Intuitively, shifting F(z) in the direction of this gradient ("Errors" Increasing) would likely worsen the model's performance, analogous to how traditional gradients indicate a direction in parameter space that increases the loss [55). Therefore, it is natural that we should calibrate F(x) in ("Errors" " Reducing).
the direction of negative gradient-
V F ((
Definition 2. Gradient set Diaiin Drrain = ((z. - F(( (F (F 9))(() y) E Derain is derived from train set Devain and stores training sample features and their corresponding negative gradients of the loss function with respect to the output of F(x). The only difference between Derain and Derain lies in the labels, which are replaced by the negative gradients in D while r" induces the same leaf node partitioning of the feature space for both datasets.
After obtaining the Gradient Set D Draiin we apply a learnable Gradient Net (z brainn r*) model the mapping between x and the corresponding negative gradient within D. Diaain where each leaf node (rr) corresponds one specific mapping function (r(:: 9:) x - (((x) The training process for Φi is as follow:
min Σ
| 0t) - (-V F(a) (F))) 1)
(r-)
where Φi could be easily implemented by conventional machine learning model, such as CART 30 and Φi over different leaf nodes are trained separately.
At inference time, given x from Drest, the prediction of Gradient Net & could be produced as a sample-specific error correction vector A,. The final prediction F*(x)) could be obtained by adding A, to F(x), that moves F(x) in the direction of negative gradient ("Errors" Reducing) by one step:
Node(r")
A2 = 17 Φ(x) Frais r") = 17
(1(1; I(x E L((r)),
(5)
F*)) = F(z)+Am
(6)
where T(.) is an indicator function determining if of interest would be assigned leaf node (r*) based on rule r", E IR, is the hyperparameter similar to learning rate controlling the step size, and
A, approximates -17* * V F()) ((F()))))))))))) v). We now explain the proposed error correction strategy by giving a closer look at Eq. 6
Proposition 1 Let E (2(x), v)1 denote the expected loss, where F(x) = i ΣK fx(x) Tk) and each fa corresponds decision tree rule Ts from the expert-derived rule set R fru 1==1-
KK

---

Given prompt p that contains R. the expected loss E (FF(F()), y)] could be decreased by querying the LLM with p to generate refined rule r", which enables approximation of a cample-speciffe error correction vector A to guide prediction F F(x)) in the direction of "errors" reducing.
The proof of Proposition 1 is provided in Appendix A.5 To reduce the expected loss, we can approximate and apply negative gradient-based error correction vector Ag to F(x).
4.3 Framework overview and discussion
Framework overview. In terms of training stage, the procedure begins by training a set of different decision tree experts (using CART for each expert) and obtaining the ensemble predictions based on Eq. 2 We then extract the corresponding decision tree rule set R. The set of rules is used to construct prompt p for LLM, which returns an improved rule r (Eq. B). Subsequently, we compute the negative gradient for each training sample to construct the Gradient Set Train and then train the
Gradient Net & based on Dyvia and r" (Eq. 4p. During the test stage, we can achieve the preliminary prediction result Fxx) for each test sample r based on the random forest; and also compute the sample-specific error correction vector A by feeding the test sample to Gradient Net. Now the final prediction can be corrected using Eq. The complete training and inference pipeline is summarized in Algorithm 1 of Appendix A.3
Discussion. To sum up, fed with the expert-derived rules R. LLM prompted to summarize multiple rules and redesign a new rule r", which thus considers the inherent relationships and interactions among multiple independent rules. Benefiting from the rule-based prompt, we neither need to fine- tune LLM nor expose the serialized samples into LLM. Due to the powerful reasoning ability of LLM, the refined rule r" that summarizes multiple rules is more effective and also convenient for us to use. For example, we can not only use the rule to make decision but also to partition the feature space of training set. The latter means that we can build a set of disjoint leaf nodes, where samples grouped within the same leaf node exhibit greater statistical similarity and thus can be used to train network. Here, we use the samples in the same leaf node to model the mapping between sample and the corresponding negative gradient, which is usually easier than directly fitting the relationship between samples and the labels. Our proposed DeLTa enables a principled integration of expert knowledge and LLM-based rule synthesis for enhanced performance, which paves new way for LLM-based tabular data learning.
5 Experiments
Datasets. We consider variety of supervised tabular prediction tasks with heterogeneous features, including binary classification, multiclass classification, and regression. Specifically, the tabular datasets include: Blood (BL) (56), Credit (CR) [57), Car (58), Bank (BA) [59]. Adult (AD) (60), Jan- nis (JA) [T]. Cpu act (CP) IT) Credit_ reg (CRR) [II] California housing (CA) [6], House 16H (HO) (. Fried (FR) 622 Diomand (DI) 633. The dataset properties and data pre-processing
details are summarized in Appendix I Following previous studies [5], we use Accuracy (higher is better) to evaluate binary and multiclass classification tasks, Normalized Root Mean Squared Error (NRMSE) (lower is better) to evaluate the regression tasks.
Baselines and implementation details. We conduct a comparative analysis between DeLTa and other prominent methods in the field of tabular prediction. Specifically, we include LLM-bbsed methods such Ta 20 LIFT 19]. TP-BERTa (2T) GTL 22 P2T 23 FeatL_LM 24 For LIFT, we employ two versions, i.e., fine-tuning L.LMs (LIFT) and in-context learning (LIFT-ICL) versions, as provided in their original paper. In addition, we compare DeLTa with non LLM-based methods,
including KNN (64), CART [30]. MLP (65), Random Forest [51). XGBoost [8). CatBoost [10],
FT-Transformer [5). TabPFN
442. ModernNCA [38] More details about the baseline models can be found in Appendix A.2 Implementation details of DeLTa including myper-parameed
are provided | For LLM usage, DeLTa adopts GPT-4o as its LLM backbone, yet is designed to be agnostic to the choice of LLMS (see Appendix A.6] for more results). To prevent LLMs from generating task-irrelevant output, we query LLM times by default and use the average. In replicating baselines, GPT-40 is utilized for in-context learning. For baselines require LLM fine-tuning, we employ the L.LMs used in their original paper: TabL.LM uses TO 46;; we use the API provided by OpenAl to perform black-box GPT-3.5 fine-tuning for LIFT, where the fine-tuning
in Appendix [A.3]
7

---

Table 2: Test Accuracy (1) performance DeLTa and LLM-based baseline methods classification tasks. "# Num" indicates number of training samples. "Tab" indicates whether the LLM requires access tabular samples. "Extra" indicates whether the method needs to use extra data samples outside the training set. "Feat" indicates whether the feature names are needed. "Tune" indicates whether LLM fine tuning is needed.
N Method usod)
Addiiooal requirements
Dulasets
Tab
Extra
Cur
BA
AD
AlI AlI
TaBLLM (TO)
0.761
0.737
0.545
0.906
0.961
0.691
0.501
LIFT
0.689
0.729
0.825
AlI
TP-RERT (RoBFRTR)
0.761
8.730
0.336
0.916
0.844
AII
GTL (LLaMA2)
NAA
Ni
NiA
N/A
NIA
N/A
NIA
AlI
LIFT-ICL, (CPT-46)
NIA
N/A
N7A
N/A
N/A
N/A
NIA
(GPT-40)
N/A
N/A
AlI
FRaILLM (GPT-MGGPTOO
0.768
0.200
0.589
0.887
0.842
0,540
0.721
AlI
DeL.a (GPT-4o)
0.829
0,783
0.836
0.908
0,866
0.705
0,822
256
TALL (T))
0.344
0.600
0.471
0.785
0.787
0.323
0.575
256
LIFT (GPT-3.5)
0.651
0.682
0.431
0.734
0.773
0.398
0.610
256
TR-BERT: (RoBERTH)
0,677
0,408
0,77
0,464
256
256
256
LIFTICL (GPT-40)
(TMA2))MA2)
P2T (GPT-40)
XX
NIA
NW
WY
WWN
XXX
N/N
NN
NA
N/A
N/
VAN
NIA
NYA
NVA
Feal (GPT-40)
256
DeL.a (GPT-4)
0.732
0.713
0.736
0.767
0.778
0.476
0.700
TaBLI
0.245
0.673
0.386
0.612
0.785
0.268
0.91
64
LIFT (GPT-3.5)
0.645
01643
0.500
0.728
0.748
0.430
0.615
64
TP-BEKI (RoBEKTa)
0.648
01666
0.419
0,840
0.761
0.414
0.621
GTT, (.MA2))
0.512
0.719
LIFT-CII (GPT-4))
0.348
0.538
0.376
0.460
8.63
0.251
0.44
64
PET (GPT-44)
0368
0.544
0.369
0.634
0,245
0.448
64
Fext LM (GPT-40)
0.407
0.643
0.508
0.723
0 740
0.450
0.678
DeLTa (GPT-44)
CA
C3R
CRR
CA
ERR
CA
ERR
0 02040600
Coota de
FR
FR
data
(c) 128-shot
(d) 64-shot
LIFT
TP-3ERT
GTL
LIFT- CL
P2T
DelTa [Ours)
Figure 3: Test NRMSE (4) performance of DeLTa and LLM- baseline methods on regression tasks.
method is suggested by the original paper; TP-BERTa employs RoBERTa [66 : GTL uses the 13B
version of LLaMA 2 (67. The experiments are run on NVIDIA A100-PCIE-40GB GPU.
5.1 Main Results
Comparison with LLM-based methods. We evaluated the performance of DeLTa against main- stream LLM-based baseline methods tabular classification tasks, as illustrated in Table 2 In addition to standard prediction tasks under full-data regimes, we also provide results under low-data regimes (few-shot learning), some of these LLM-based baseline methods are constrained by the limited input length of LLMs and are thus restricted to low-data settings. Following prior work (24), simulate the few-shot setting by randomly sampling fixed number of training examples, where the number shots denotes the total number of selected samples. Considering some of the I.L.-- based baseline methods are applicable to regression tasks, we further evaluate DeLTa on regression under both full-data and low-data regimes settings, as illustrated in Fig. 3 Overall, DeLTa achieves the highest average performance in all settings, including classification and regression, in both full and low-date regimes. While DeLTa is primarily designed for full-data tabular prediction, our results demonstrate that it also performs competitively in low-data scenarios. Furthermore, compared to existing LLM-based tabular prediction methods, the proposed DeLTa requires no additional require- ments for LLMs as illustrated in Table also aligns with our claims that we have solved the two inherent issues for prior M-aee approaches: (i) the proposed DeLTa avoids reliance on column names and tabular data serialization, and thus will protect sample-leve privacy information, and (ii) rather than directly using LLMs to generate label predictions, DeLTa utilizes the powerful reasoning

---

Table 3: Runtime in seconds of DeLTa and other LLM-based baseline methods for the training and inference phase, conducted on Adult dataset. "# Num" indicates number training samples. Thesee methods require querying LLMs for each test sample. These methods do not require querying LLMs at inference phase.
Nom
Stage
TabLLMr
LIFIT
TP-BEKIa
GILT
LIFRICLT
P2TT
FeatLMM
DeL.la" (Ours)
All
Irain Inferenze
1773/1.37 179.21
153689.1
90169.48
1319,35
1.38
NIA NIA
N/A N/A
N/A
N/A
1231,22
0.14
35.20
0,09
64
Trin Inference
268.61
170.42
89684.76
191.75
316.40
5.01
153501 OZ
N/A
176726.13
N/A
200258.24
N/A
1047.55
0.14
23,09
0
Classification
Classification
Regression
Regression
0.82
0.50
1 0.00
0.55
Rank (-)
Accurecy
0.78
NRWSE
0.50
0.76
0.45
0.40
Figure 4: Test performance of DeLTa and non-L.M baseline methods on classification and regression tasks. Here, "RF" denotes Random Forest, "FT-T" denotes Fr-nrfnsformer, "MNCA" denotes ModernNCA.
ability of LLMs to enhance decision trees, enabling effective tabular prediction in both full-data and low-data regimes without requiring L.M fine-tuning.
Computational efficiency. We compare the computational cost between ours and LLMM-based methods in Table 3 The results show that our proposed DeLTa is computationally efficient in both training and inference stages. This efficiency stems from key properties: (i) DeLTa utilizes the reason ability of LLM without fine-tuning LLM, (ii) DeLTa avoids querying LLMs to generate predictions for individual samples. Instead, it only needs to query LLM via API to generate one refined decision tree rule for one dataset, enabling significantly more efficient use of LLM resources.
Comparison with conventional methods. We further compare the performance of DeLTa and other conventional tabular models on full data regimes averaged over all datasets, as illustrated in Fig. 4 Full results are provided in Appendix A.8 The results show that DeL.Ta also performs well compared with conventional tabular models both classification and regression tasks.
5.2 Further analysis
Ablation study. We further conduct ablation study to demonstrate the effectiveness of key components of DeLTa, as illustrated in
Table 4: Ablation study on the different components in DeLTa; see full results in Appendix A. "RR" denotes the gule refinement process, and "EC" denotes the error correction process.
T by comparing DeL.a against three variants: (i) DeLTa w/o
"RR", that replaces the LLM generated r* used in Gradient Net 6 train. r") with r E R derived from Random Forest F(x),
(ii) DeLTa w/o "EC", that directly obtains the label prediction for test x by utilizing [(x ((x]/rrrinn Drrain: r") (Eq. 1 with the refined rule r* (iii) Random Forest, which makes decision with F(x) in Eq. The results show that the decision tree rule refinement via LLM is important, and the error correction strategy could further enhance tabular prediction performance.
Variants
CR
CA
Del.Ta (Ours)
0.7829
0.3507
w/o "RR"
0.7257
0,5443
who "EC"
0.7657
0.4016
Random Forest
0.7371
0.5057
Additional results. Fig. 5 shows the la- bel predictions of ours: F(x) + Ar and ours w/o Ag: F(x). We can observe that two categories of samples are overlapped in some regions. F(x) struggles to distin- guish them, but DeLTa could correct the prediction errors of F(x) to enhance the prediction for such complex data patterns.
Loc
leeat
(a) Ground Truth
(b) Predicted by Ours wio
[c] Predicted by Ours
Figure 5: Visualization of label prediction of DeL.Ta w/ and w/o error correction vector A, BA dataset.
We incorporate the sensitivity analysis for hyperparameters Appendix A.11 and analysis regarding the LLM-refined rule can be found in Appendix A.12
Additional insights

---

Conclusion
In this paper, we propose DeLTa, decision tree enhancer with LLM-derived rule for tabular prediction, to solve the two challenges w.r.t data perspective and model perspective in prior LLM- based approaches for tabular prediction. DeLTa avoids reliance on feature names and tabular data serialization, and it utilizes the powerful reasoning ability of LLMs to enhance decision trees, enabling effective tabular prediction both full-data and low-data regimes without requiring LLM fine-tuning. The empirical results demonstrated the effectiveness of DeLTa for tabular prediction tasks. Our work can shed some light on developing better algorithms for similar tasks.
Acknowledgments and Disclosure of Funding
The authors would like thank the anonymous referees for their valuable comments. In this work, Hangting Ye. Jinmeng Li, Dandan Guo and Yi Chang are supported by the National Key R&D Program China under Grant (No. 2023YFF0905400) and the National Natural Science Foundation of China (No. 623B2043, No. U2341229, No. 62306125).
Part of this work was carried out during Dandan Guo's visit to King Abdullah University of Science and Technology (KAUST).
References
[1] Mikel Hernandez, Gorka Epelde, Ane Alberdi, Rodrigo Cilla, and Debbie Rankin. Synthetic data generation for tabular health records: A systematic review. Neurocomputingg 493:28 45, 2022.
[2] Dominick L Frosch, David Grande, Derjung M Tarn, and Richard L Kravitz. A decade of controversy: balancing policy with evidence the regulation of prescription drug advertising. American Journal of Public Health, 100(1):24-32, 2010.
[3] Samuel A Assefa, Danial Dervovic, Mahmoud Mahfouz, Robert E Tillman, Prashant Reddy, and Manuela Veloso. Generating synthetic data in finance: opportunities, challenges and pitfalls. Proceedings of the First ACM International Conference Al in Finance, pages 1-8, 2020.
[4] Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelezyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey. IEEE transactions on neural networks and learning systems, 2022.
[5] Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances in neural information processing systems, 34:1 18932- 18943, 2021.
[6] Hangting Ye, Wei Fan, Xiaozhuang Song, Shun Zheng, He Zhao, Dan dan Guo, and Yi Chang. PTaRL: Prototype-based tabular representation learning via space calibration. In The Twelfth International Conference Learning Representations, 2024.
[7] Hangting Ye, He Zhao, Wei Fan, Mingyuan Zhou, Dan dan Guo, and Yi Chang. DRL: Decom- posed representation learning for tabular anomaly detection. In The Thirteenth International Conference on Learning Representations, 2025.
[8] Tianqi Chen and Carlos Guestrin. Xgboost: scalable tree boosting system. Proceedings of the 22nd aem sigkdd international conference knowledge discovery and data mining, pages 785-794, 2016.
[9] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie Liu. Lightghm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30, 2017.
[10] Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. Advances neural information processing systems, 31, 2018.
10

---

[11] Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree- models still outperform deep learning on typical tabular data? Advances in neural information processing systems, 35:507-520, 2022.
[12] Wei Fan. Kumpeng Liu, Hao Liu, Pengyang Wang, Yong Ge, and Yanjie Fu. Autofs: Auto- mated feature selection via diversity- aware interactive reinforcement learning. In 2020 IEEE International Conference on Data Mining (ICDM), pages 1003-1013, IEEE, 2020.
[13] Wei Fan, Kunpeng Liu, Hao Liu, Yong Ge, Hui Xiong, and Yanjie Fu. Interactive reinforcement learning for feature selection with decision tree in the loop. IEEE Transactions on Knowledge and Data Engineering, 35(2):16241666, 2021.
[14] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[15] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800 04005, 2018.
[16] Hangting Ye, Peng Wang, Wei Fan, Xiaozhuang Song, He Zhao, Dandan Guo, and Yi Chang. Deep tabular representation corrector. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-15, 2025.
[17] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[18) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 2rxiiv2303.088744, 2023.
[19] Tuan Dinh, Yuchen Zeng, Ruisu Zhang, Ziqian Lin, Michael Gira, Shashank Rajput, Jy-yong Sohn, Dimitris Papailiopoulos, and Kangwook Lee. Lift: sanguage-interfaced fine-tuning for non-language machine learning tasks. Advances in Neural Information Processing Systems, 111116631778.. 2022.
[20] Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. Tablim: Few-shot classification of tabular data with large language models. In International Conference on Artificial Intelligence and Statistics, pages 5549-5581. PMLR, 2023.
[21] Jiahuan Yan, Bo Zheng, Hongxia Xu, Yiheng Zhu, Danny Chen, Jimeng Sun, Jian Wu, and Jintai Chen. Making pre-trained language models great on tabular prediction. In The Twelfth International Conference on Learning Representations, 2024.
[22] Xumeng Wen, Han Zhang, Shun Zheng, Wei Xu, and Jiang Bian. From supervised to generative: novel paradigm for tabular deep learning with large language models. In Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pages 3333-3333, 2024.
[23] Jachyun Nam, Woomin Song, Seong Hyeon Park, Jihoon Tack, Sukmin Yun, Jaehyung Kim, Kyu Hwan Oh, and Jinwoo Shin. Tabular transfer learning via prompting LLMs. In First Conference Language Modeling, 2024.
[24] Sungwon Han, Jinsung Yoon, Sercan O Arik, and Tomas Pfister. Large language models can automatically engineer features for few-shot tabular learning. In International Conference on Machine Learning, pages 17454-17479. PMLR, 2024.
[25] Jiahuan Yan, Jíntai Chen, Chaowen Hu, Bo Zheng, Yaojun Hu, Jimeng Sun, and Jian Wu. Small models are LLM knowledge triggers for medical tabular prediction, In The Thirteenth International Conference Learning Representations, 2025.

---

[26] Xi Fang, Weijie Xu, Fiona Anting Tan, Jiani Zhang, Ziqing Hu, Yanjun Qi, Scott Nickleach, Diego Socolinsky, Srinivasan Sengamedu, and Christos Faloutsos. Large language models (Ilms) on tabular data: Prediction, generation, and understanding-a survey. arXiv preprint arXiv:2402. 1744,, 2024.
[27] Biwei Yan, Kun Li, Minghui Xu, Yueyan Dong, Yue Zhang, Zhaochun Ren, and Xiuzhen Cheng. On protecting the data privacy of large language models (IIms): A survey, arXiv preprint arXiv:2403.05756 2024.
[28] Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. survey on large language model (Ilm) security and privacy: The good, the bad, and the ugly. High-Confidene Computing, page 100211, 2024.
[29] Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efffcient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403. 2024.
[30] Wei-Yin Yin Loh. Classification and regression trees. Wiley interdisciplinary reviews: data mining and knowledge discovery, 1(1):14-23, 2011.
[31] Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi. Den v2: Improved deep cross network and practical lessons for web-scale learning to rank systems. Proceedings of the web conference 2021, pages 1785-1797, 2021.
[32] Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. Advances in neural information processing systems, 30, 2017.
[33] Hangting Ye, Zhining Liu, Xinyi Shen, Wei Cao, Shun Zheng, Xiaofan Gui, Huishuai Zhang, Yi Chang, and Jiang Bian. Uadb: Unsupervised anomaly detection booster. In 2023 IEEE 39th International Conference on Data Engineering (ICDE), pages 2593-2606. IEEE, 2023.
[34] Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. Autoint: Automatic feature interaction learning via self-attentive neural networks. Proceedings of the 28th ACM international conference on information and knowledge management, pages 1 1161-1170, 2019.
[35] Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106.0/1 0442, 2021.
[36] Sercan Ô Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In Proceed-
ings of the AAAI conference on artificial intelligence, volume 35, pages 6679-6687, 2021.
[37] Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.06678 2020.
[38] HJ Ye, HH Yin, DC Zhan, and WL Chao. Revisiting nearest neighbor for tabular data: A deep tabular baseline two decades later. ICLR, 2(3):4, 2025.
[39] Jannik Kossen, Neil Band, Clare Lyle, Aidan N Gomez, Thomas Rainforth, and Yarin Gal. Self-attention between datapoints: Going beyond individual input-output pairs in deep learning. Advances in Neural Information Processing Systems, 34:28742-28777 2021.
[40] Yury Gorishniy, Ivan Rubachev, Nikolay Kartashev, Daniil Shlenskii, Akim Kotelnikov, and Artem Babenko. Tabr: Tabular deep learning meets nearest neighbors in 2023. arXiv preprint arXiv:2307. 14338, 2023.
[41] Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, and Mahsa Shoaran. Xtab: Cross-table pretraining for tabular transformers. arXiv preprint arXiv:2305.. 06090, 2023.
[42] Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirmeisterr, and Frank Hutter. Accurate predictions on small data with a tabular foundation model. Nature, 637(8045):319-366 2025.
12

---

[43] Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. TabPFN: A transformer that solves small tabular classification problems in second. The Eleventh International Conference on Learning Representations, 2023.
[44] Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang, and Jun Wang. Ds-agent: Automated data science by empowering large language models with case-based reasoning. arXiv preprini wrXiv:2402.174555 2024.
[45] Siyuan Guo, Huiwu Liu, Xiaolong Chen, Yuming Xie, Liang Zhang, Tao Han, Hechang Chen, Yi Chang, and Jun Wang. Optimizing case-based reasoning system for functional test script generation with large language models. Proceedings of the 31 ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pages 4487-4499 2025.
[46] Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang. Mike Tian-Jian Jiang, Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022.
[47] Hariharan Manikandan, Yiding Jiang, and J Zico Kolter. Language models are weak learners. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[48] Jaehyun Nam, Kyuyoung Kim, Seunghyuk Oh, Jihoon Tack, Jaehyung Kim, and Jinwoo Shin. Optimized feature generation for tabular data via Ilms with decision tree reasoning. Advances in Neural Information Processing Systems, 37:92352-92388 2024.
[49] Tianping Zhang, Zheyu Aqa Zhang, Zhiyuan Fan, Haoyan Luo, Fengyuan Liu, Qian Liu, Wei Cao, and Li Jian. Openfe: Automated feature generation with expert-level performance. In International Conference on Machine Learning, pages 41 41880-41901. PMLR, 2023.
[50] Tin Kam Ho. The random subspace method for constructing decision forests. IEEE transactions on pattern analysis and machine intelligence, 20(8):832-844, 1998.
[51] Leo Breiman. Random forests. Machine learning, 45:5-32, 2001.
[52] TONG ZHANG and BIN YU. Boosting with early stopping: Convergence and consistency. Annals of statistics, 33(4):1538-1579, 2005.
[53] Jerome H Friedman. Greedy function approximation: gradient boosting machine. Annals of statistics, pages 1189-1232, 2001.
[54] Jerome H Friedman. Stochastic gradient boosting. Computational statistics & data analysis, 38(4):367-37,, 2002.
[55] Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint 1rX1v:1609.04777, 2016.
[56] I-Cheng Yeh. Blood Transfusion Service Center. UCI Machine Learning Repository, 2008 DOI: http:///ooooooo//o.24/////CCC
[57] Hans Hofmann. Statlog (German Credit Data). UCI Machine Learning Repository, 1994. DOI: htts://doo.org/1..4452//
[58] Marko Bohanec and Vladislav Rajkovic. Knowledge acquisition and explanation for multi- attribute decision making. In 8th intll workshop on expert systems and their applications, pages 59-78. Avignon France, 1988.
[59] Sérgio Moro, Cortez, and Paulo Rita. data-driven approach to predict the success of bank telemarketing. Decis. Support Syst., 62:22-31, 2014.
13

---

[60] Ron Kohavi et al. Scaling up the accuracy of haive-bayes classifiers: A decision-tree hybrid. In Kdd, volume 96, pages 202-207, 1996.
[61] Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics Probability Letters, 33(3):291-297, 1997.
[62] Leo Breiman. Bagging predictors. Machine learning. 24:123-144 1996.
[63] Kuan-Yo Yu Chen, Ping-Han Chiang, Hsin-Rung Chou, Ting- Wei Chen, and Tien-Hao Chang. Trompt: Towards better deep neural network for tabular data, arXiv preprint arXiv:2305. 18446, 2023.
[64] Thomas Cover and Peter Hart. Nearest neighbor pattern classification. IEEE transactions on information theory, 13(1):21-27, 1967.
[65] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):446-444, 2015.
[66] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692, 2019.
[67] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0922 2023.
[68] Han-Jia Ye, Si- Yang Liu, Hao-Run Cai, Qi-Le Zhou, and De-Chuan Zhan. A closer look at deep learning on tabular data, arXiv preprint (rXiv:2407.00955 2024.
[69] Si- Yang Liu, Hao-Run Cai, Qi-Le Zhou, and Han-Jia Ye. Talent: tabular analyties and learning toolbox. arXiv preprint arXiv:2407.04057, 2024.
[70] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the claims made in the abstract and introduction accurately reflect the paper's contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, discussed the limitations of the work Appendix A.14.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification: Yes, we provide the full set of assumptions and complete (and correct) proof in Appendix A.5.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, we provide the information needed reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
16

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Yes, we provide the access to the data and code with instructions to reproduce the main experimental results at htttps://ggthtii wttplau...ooooo// Coom/HangtingeeDDalTT
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
public/guides/ Staalees//ooeSSusmsssiioeeoe
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
Coaee//ddeeeSbmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, specify all the training and details necessary to understand the results.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report standard deviation in Appendix A.7.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, we provide information on the computer resources needed to reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics hattps://neurip::::: http://ss/aarrpp...ccc/a///////a
Answer: [Yes]
Justification: Yes, the research conducted in the paper conform with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, we discuss the both potential positive societal impacts and negative societal impacts of the work performed Appendix A.16.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, the creators original owners of assets used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, batasets.papersstt paperswi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, new assets introduced in the paper well documented and is the documen- tation provided alongside the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: Yes, we describe the usage of LLMs.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: //appp:///eurps..... http///ripscccooooeeeereee/////p cc//oonferences/22255/LI. for what should or should be described.
21

---

20640 22784 A Technical Appendices and Supplementary Material
Technical appendices with additional results, figures, graphs and proofs may be submitted with the paper submission before the full submission deadline (see above), as separate PDF in the ZIP file below before the supplementary material deadline. There is no page limit for the technical appendices.
A.1 Datasets details
We consider a variety of supervised tabular prediction tasks with heterogeneous features, including classification and regression. Specifically, the tabular datasets include: Blood (BL), Credit (CR), Car, Bank (BA), Adult (AD), Jannis (JA), Cpu_act (CP), Credit_reg (CRR), California_house House_16h (HO), Fried (FR), Diomand (DI). The dataset properties are summarized in Table 5 Our primary objective was to evaluate DeL.aas effectiveness in the context of LLM-based tabular prediction, and to ensure fair comparison with existing methods in this category. Accordingly, the majority of our classification datasets were selected based their widespread use in recent LLM- based tabular learning studies such as TabLLM and FeatLLM, both which primarily focus on classification tasks. To extend our evaluation beyond classification, also incorporated regression datasets from the TALENT benchmark [68 69], which has recently gained attention for assessing tabular regression performance. To ensure fair comparisons, adhere to identical preprocessing procedures following TALENT benchmark (69 for each dataset.
(CA),
Table 5: Tabular dataset properties. "#objects" indicates the number of samples in the dataset. "#num. features" indicates the number of numerical features, and "#cat. features"' indicates the number of categorical features.
BL
CR
Car
BA
AD
JA
CP
CRR
CA
HO
FR
DI
#objects
748
1000
1728
45211
48842
83733 8192
16714
40768
53940
Anum. features
#eat, features
0
13
6
8
0
0
0
0 NRMSE
0 NRMSE
0
3 NRMSE
metric
Acc.
Acc.
Acc.
Acc.
Acc.
Acc.
NRMSE
NRMSE
NRMSE
#classes
2
A.2
Baseline models details
We include LLM-based methods such as TabLLM, LIFT, LIFT-ICL, TP-BERTa, GTL, P2T, FeatLLM:
(1) TabLLM employs TO the base LLM and investigates the performance of fine-tuned TO model in very few-shot scenarios; (2) LIFT investigates the performance of fine-tuned GPT models on tabular data; (3) LIFT-ICL another version of LIFT provided the original paper, that utilizes in-context learning without fine-tuning by providing a few training examples; (4) TP-BERTa introduces tabular-spccific tokenization scheme, pretraining single language model across multiple tabular datasets. Given target tabular datasets, the pretrained language model should be further fine-tuned; (5) GTL pretrains LLaMA across multiple tabular datasets to achieve zero-shot and in-context learning on target tabular dataset; (6) P2T leverages extra unlabeled samples outside the training set as demonstrations to conduct in-context learning: (7) FeatLLM prompts LLMs with few training examples to generate new features to replace original features and then trains linear models for few-shot classification tasks. replicating baselines, GPT-4o utilized for in-context learning. For baselines require L.LM fine-tuning, we employ the L.LMs used in their original paper: TabI_MM uses TO; we use the API provided by OpenAl to perform black-box GPT-3.5 fine-tuning for LIFT, where the fine-tuning method is suggested by the original paper; TP-BERTa employs RoBERTa; GTL uses the 13B version of LLaMA The implementation of these methods is based on their official open-source code releases.
In addition, we compare DeLTa with non LLM--bsed methods, including KNN, CART, MLP, Random Forest, XGBoost, CatBoost, ResNet, FT-Transformer, Saint, TabPFN and ModernNCA. Among them, (1) KNN classifies or predicts sample based on the majority vote or average of its nearest neighbors in the feature space; (2) CART is an implementation for the simplest decision trees; (3)
MI.P is type of feedforward neural network composed of multiple layers; (4) Random Forest, (5)
XGBoost and (6) CatBoost are the variants of decision tree-based methods; (7) ResNet is deep
22

---

neural network architecture that introduces skip connections to enable the training of very deep networks; (8) "T-Trassformer is a boken-based method which transforms features to embeddings and applies series of beentioo-bbsedd transformations to the embeddings; (9) Saint is token-based method that leverages row and column attention mechanisms for tabular data; (10) TabPFN generates synthetic dataset with diverse distribution to pretrain a model, and then leverages the training samples of target dataset for inference. We use the version 2 of TabPEN since it can be applied to both classification and regression datasets; (11) ModemnNA makes predictions based on the relationships with neighbors a learned embedding space, The implementation of these methods based on the code releases from TALENT benchmark [69].
official
A.3 DeLTa details
Algorithm 1 DeLTa training and inference workflow.
Input: Training set Dtrain, test set Drest (without accessing to label), number of decision tree experts K, API of LLM;
Initialize source decision tree rule set R []:
for k 1 to K do // This loop can be executed in parallel.
Train one decision tree expert fk on Dirain with Eq. 1;
Extract the corresponding rule r of fx:
Update R < R Urk
end for
Obtain the ensemble prediction of decision tree experts F(x) with Eq. 2:
Form prompt based on R;
Query LLM with prompt y and generate the new rule r" with Eq. 3; y) E D Dramm
Calculate Gradient Set D Dirain = (Ir -V r(r) (FF()
Train Gradient Net & based on Gradient Set with Eq. 4;
for in Drest do // Without accessing to label. And this loop can be executed in parallel.
Estimate error correction vector A, = 17 * (( r*) with Eq. 5;
Computing final prediction F" (x) = F(x) A2 with Eq. 6;
end for
Return: Final prediction {F* (z))(x Drest).
The DeLTa architecture remains consistent across all datasets. Specifically, the F(x) is produced by the output of Random Forest. For LLM usage. DeLTa adopts GPT-4o via API as its LLM backbone, yet is designed to be agnostic to the choice of LLMs (see Table Table more results). To prevent LLMS from generating task-irrelevant output, we query LLM 10 times by default and use the average. DeL.T needs to construct Gradient Set Drain = {zz - [f (F()), v)) (x, v) E Dtrain]. which stores training sample features and their corresponding negative gradients of the loss function with respect to the output of F(x). Here, is a smooth loss function (e.g., mean squared error for regression or cross-entropp for classification). And - [ v) R for regression, - (f 3) E R for binary classification, and V (F()) E R for multiclass classification. where is the number of classes. The Gradient Net ø contains learnable leaf node-specific mapping function ø1, where Φi is implemented by CART for classification and TabPFN for regression, and Φr over different leaf nodes are trained separately. The number of leaf node for LLM generated rule r' will less than 30, and different Φi can be trained in parallel, which will result in low runtime cost as illustrated in Table 14 The training and inference pipeline is summarized in Algorithm
A.4 Example prompt for refining decision tree rules
We leverage LLM to refine the decision tree rule set R to generate a better rule r", where samples grouped within the same leaf node exhibit greater statistical similarity. Therefore, we query LLM to generate a better rule to partition the feature space into disjoint regions (i.e., leaf nodes). Specifically, the prompt p is designed to include: meta information Pmncta that describes the task objective, the extracted decision tree rule description Prute that contains R, and the requirement Prequirement for rule refinement. Please note that our approach does not require additional domain-specific expertise or semantic information, such explicit feature names detailed task background knowledge. This
23

---

makes it applicable to scenarios where such information is unavailable or incomplete. Instead of relying on the intrinsic domain knowledge tabular tasks, such as those healthcare or finance, we frame the problem purely as domain knowledge-agnostie machine learning task. Our method leverages L.LMs to reason decision tree rule set R into an rule r", rather than directly inferring from serialized tabular data, and thus will protect sample-level privacy information. The prompt example provided in Fig.
You are an expert in tabular machine learning domuin. I will provide the meta information, the CART tree rules about the prediction task. Please help design better rule for inference. information about
{
"casku type": "bincluss", "n_oun_ featuree". 5,
"m_cat_ Fasturraa. 8,
# CART tree miles (We provide the example of decision nule of one subtree from the Random Forest below:)
Tree rules:
feature._10
-0.59
feature 2 0.94
----
feature_6 <m -1.82
1-.- clase:
feature_6
1--- clare: 0.0
feature,2 0.94
featuze_8 0.53
1--- clase: 1.0
4.81
feature._10 -0.59
1--- feature_0 <m -0.74
I--- class:
feature_ -0.74
I... feature_ 3 0.80
reature_2 2 <- 0,84
class:
feature_2
featuro_10 1.89
I--- class: 0.0
1--- feature_3 0.80
1-.. class: 1.0
CART tree rules end
Based above information, please learn
evolving process and help me design a better what cart used for inforence
to achieve higher performance. Please not just copy. please reline these rules and create a new better one. The new rule aims to divide the training space into several regions, where each region denoted by unique leaf node id. The number of leaf nodes should more than Please return the dict format rule, format should be strictly like:
self. cree
"Soature":
"thrwahold": -0.78,
"operator": *ea",
'icrt": ("id": "ioat_1"}, "rightt:
"Ceature": 7,
"threshold"; 0.46,
"operator"; he-
"lefe": ("id": "leaf_2")
*right': ("id": "1a)
Please note that cach Icaf id node can only appear onoe, for example, "id": "Leaf 1 can only appear once. Thus you only need to return
predictions.
Figure 6: An example the prompt template for the adult dataset. Here, r denotes the maximum number of leaf nodes among all the provided decision trees.
24

---

A.5 Theoretical analysis
Proposition 2 Let B (L(F(z)) y)] denote the expected loss, where F(x)= Es (a train Tk) and each fK corresponds Given prompt that contains R, the expected loss E (F((F))), v) could be decreased by querying the LLM with p to generate refined rule r", which enables approximation of a sample-specific error correction vector Dr to guide prediction F F(x) in the direction of "errors" reducing.
K decision tree rule T& from the expert-derived rule set R fri T
Proof. The the expected loss E [X(((x)) y) after calibration is formulated as:
([ = [f(F(x) A.. v)]
* R E [F(F(x)) + ((O))
E [I(F(E)))) ((()) [(F(e))) (7)
E
(F(((()))))
where [[<(F(x) + A.. v)] = E [F((()))))))))) + + V ( y)] is given by Taylor first- order expansion and I is a convex loss function (e.g.. mean squared error or cross-entropy). Error correction vector Ar = fy* ( D.vain r") approximates the -1}* V (F(rr) v) in the direction of "errors" reducing according to Eq. in the original paper and 1) E R4 the hyperparameter. To reduce the expected loss, we can approximate and apply negative gradient-basee error correction vector A, F(x).
A.6 Varying LLM backbones
To investigate the impact of using different LLMs in our framework, given their distinct prior knowl- edge and reasoning abilities from being trained on various text corpora, we measured performance using only GPT-4o but also the Qwen3-32B (open-source) as a backbone for comparison. The results in Tad Table indicate that while certain tasks show greater improvement with specific LLMs, the average performance similar with different LLMs.
Table 6: Effects of various LLM backbones: GPT-4o vs. Qwen3-32B on classification tasks 1 in full-data and low-data regimes. "# Num indicates the number of training samples.
Num
Various LLM backbones
B
CRT
Car
BA
AD
JA 1
Average
All
DeL.Ta (Qwen3 12B)
0,821
0.766
0,332
0.907
0.867
0,703
0.816
Del.Ta (GPT.4g) (Ours)
0.783
0.908
0.868
256
DeL.a (Qwen3 32B)
0.741
0.707
0.750
0.765
0.778
0.474
0.703
256
Del.Ta (GPT-4a) (Ours)
0.732
0.713
0.736
0.767
0.778
0.476
0.700
128
DeLTa (Qwen3-32B)
0.744
0.687
0.679
0.749
0.778
0.456
0,682
128
DeLTa (GPT-4o) (Ours)
0,746
0.683
0,722
0.749
0.774
0.457
0.688
64
DeLTa (Qwen3-32B)
0.657
0.729
0.759
0.650
64
Del.Ta (GPT-4o) (Oun)
0.732
0.663
0.615
0.733
0.758
0.457
0.660
Table 7: Effects of various LLM backbones: GPT-40 vs. Qwen3-32B regression tasks 4 in full-data and low-data regimes. "# Num indicates the number of training samples.
# Num
Various LLM backbones
CPL
CRR
CA
HO1
FRJ
DIJ
Average 1
All
Del.aa (Qwen3-32R)
0.114
0.788
0.367
0.541
0.201
0.129
0.357 0.351
All
DeLTa (GPT (Ours)
0.116
0.780
0.351
0,529
0,200
0.130
256
DeLTa (Qwen3-32B)
0.203
0.854
0.606
0.811
0.358
0.293
0.521
256
DeLTa (GPT-4o) (Ours)
0.234
0.841
0.568
0.799
0.397
0.277
0.519
128
Del.Ta (Qwen3-32R)
0.206
0.876
0.709
0.940
0.474
0.348
0.592
DeLTa (GPT-4o) (Ours)
0.247
0.857
0.641
0.512
0.319
0.572
64
DeLTa (Qwen3-32B)
0.289
0.920
0.737
0.944
0,598
0.406
0.649
DeLla (GPT-40) (Ours)
0.888
0.701
0.634
0.651
25

---

A.7 Full comparison results with LLM-based baseline methods
We provide the full results of DeLTa against mainstream LLM-based baseline methods in full-data and low-data regimes in Table g to Table [1] "# Num" indicates the number of training samples. "Tab" indicates whether the LLM requires access to tabular samples. "Extra" indicates whether the method needs to use extra data samples outside the training set. "Feat" indicates whether the feature names are needed. "Tune" indicates whether L.LM fine-tuning needed.
Table 8: Comparison with LLM-based methods averaged over all classification datasets 1. Average standard deviations over all datasets given.
Additional roquirements
Classification. Num
Methods
Tab
Extra
Feat
Tume
All
512
256
128
64
16
TabLLM (TO)
0.801 012
035
0.575.0ra
0.519 .44
0.493.cs
0.413.ps2
LIFT (GPT-3.5)
0.732 051
0.625
0.610
0.650
0.615
0.556
TP-BERTa (RoBERTa)
0.789 016
0.638 029
0.649 063
0.641 a76
0.621 073
0.563 091
GTL (LLaMAZ)
N/A
N/A
NIA
N/A
0.608 os8
0.552.m7
LIF (GPT 4o)
N/A
NIA
N/A
N/A
0.424 107
0.382 081
P2T (GPT-4g)
N/A
N/A
N/A
N/A
0.448 p6n
0.304 072
FeaILLM (GPT-4o)
0.721
0.664 024
0.639
0.646 g41
0.628 047
0.570 as6
Ours (GPT-4o)
005
026
018
az2
0.660).44
Table 9: Comparison with LLM-based methods averaged over all regression datasets 4. Average standard deviations over all datasets are given.
Additional requirements
Regression, Num
Methods
Tab
Exira
Feat
Tune
All
S12
256
128
64
16
TaBLLM (TO)
N/A
N/
NIA
N/A
N/A
N/A
LIFT (GPT-3.5)
0.623
0.782 056
0.715 054
0.743
0.827 098
0.995 mos
TP-BERTA (RoRERTa)
0,403 015
0.496 011
0.582
0.642 a36
0.727 028
0.887 ,28
GTL (LL:MAZ)
NIA
NIA
NIA
N/A
0.847 092
0.938 057
(GPT-4)) 4o)
0.820 084
1,025 084
P2T (GPT-40)
V
N/A
N/A
N/A
N/A
0.935
1,042
FeaILLM (GPT-4o)
N/A
N/A
N/A
NIA
N/A
Ours (GPT-44)
0.351 .o8
0.462,
0,519 nzx
0.572
0.651 p49
0,852
26

---

Table 10: Comparison with LL-M-based methods on all classification datasets f. We also calculate the average relative improvement of DeLTa against the best baseline method (1 %).. "# Num" indicates
the number of training samples. This table serves as an extension Table g
# Num
Methods
BL
CRT
Car 1
BA 1
AD f
JA
Average
All
TabLLM (TO)
0.761
0.737
0.845
0.906
0,864
0.691
0.801
All
LIFT (GPT-3.5)
0.689
0.691
0.729
0.825
0.810
0.647
0.732
TP-BERTa (RoBERTa)
0.761
0.730
0.826
0.916
0.844
0.659
0.789
AlI
GTL (LLaMAZ)
LIFT-CLL (GPT-44)
AII
P2T (GPT-4n)
Le (GPT-4o)
0.768
0.701
0.887
0.842
0.540
0.721
AlI
DeLTa (GPT-40) (Ours)
0,829
0.783
0.836
0.908
0.868
0.705
0.822 (1 2.6%)
TabLLM
0.240
0.788
512
LIFT (GPT:3.5)
0.673
0.692
0.503
0.735
0.710
0.437
0.625
512
BE-BERTa (RoBERTa)
0.716
0.694
0.889
0.767
0.487
0.638
512
GTL (LLaMA2)
512
LIFT-ICL (GPT-4o)
512
(GPT-40)
512
FeuLLLM (GPT-40)
0.736
0.681
0.544
0.745
0.774
0.504
0.664
(GIT--4)) (Ours)
0.522
0.719 (1 8.39)
256
TaLLLM (TO)
0.344
0.690
0.471
0.785
0.787
0.373
0.575
256
LIFT (GPT-3.5)
0.651
0.682
0.724
0.773
0.398
0.610
256
TP-BERIa (RoBERT))
0.721
0.677
0.408
0.853
0.772
0.464
0.649
GTL (LLaMA2)
256
LIFT-ICL (GPT-4o)
256
(GPT-46)
256
Le (GPT-4o)
0.729
0.658
0.459
0.740
0.772
0.473
0.639
256
DeLTa (GPT-40) (Ours)
0.732
0.713
0.736
0.767
0.778
0.476
0.700
7.9%)
128
TabLLM (TO)
0.240
0.692
0.389
0.676
0.769
0.350
0.519
128
LIFT (GPT-3.5)
0.643
0.733
0.760
0.430
0.650
128
TP. BERTa (RoBERTs)
0.735
0.682
0.462
0.867
0.743
0.357
0.641
128
GTL (ILaMA2)
128
LIFT-CCL (GPT-4o)
128
P2T (GPT-4o)
FeaiLLM
0.768
128
DeL.Ta (GPT-46) (Ours)
0.746
0.683
0.722
0.749
0.774
0.457
0.688 (1 6..6%
TabLLM (TO)
0.245
0.673
0.386
0.602
0.785
0.268
0.493
64
LIFT (GPT-3.5)
0.643
0.643
0.500
0.728
0.748
0.430
0.615
(RoBERTa)
0.761
64
GTL (ILaMA2)
0.613
0.638
0.512
0.719
0.750
0.414
0.608
LIFT-ICL (GPT-40)
0.348
0.469
0.613
0.251
0.424
P2T (GPT-4o)
0.368
0.542
0.334
0.569
0.634
0.243
0.448
LM (GPT-4o)
0.697
0.643
0.508
0.723
0.749
0.450
0.628
DeLTa (GPT-40) (Ours)
0.732
0.663
0.758
0.457
0.660 (1 5.0%)
TabLIM (TO)
0.240
0.501
0.215
16
LIFT (GPT:3.5)
0.670
0.608
0.408
0.583
0.705
0.363
0.556
16
TP-REETA (RoBERTs)
0.659
0.609
0.355
0.625
0.734
0.394
0.563
GTL (LLaMA2)
0.604
0.622
0.324
0.628
0.746
0.387
0.552
16
LIFT-CCL (GPT-1o)
0.457
0.492
0.324
0.308
0.512
0.198
0,382
(GPT-40)
16
Featl 1M (GPT-4o)
0.588
0.626
0.450
0.629
0.728
0.399
0.570
DeLTa (GPT-40) (Ours)
0.682
0.638
0.730
0.379
(1 T.6%)
27

---

Table Comparison with L----based methods on all regression datasets 4. We also calculate the
average relative improvement of DeLTa against the best baseline method ( %). "# Num" indicates
the number of training samples. This table serves as an extension Table 9
# Num
Methods
CP1
CRR
CA
HOI
FRI
D11
Average L.
TabLLM (TO)
LIFT (GPT-3.5)
0.379
0.983
0.629
0.902
0.518
0.328
0.623
TP-BERTa (RoBERTa)
0.138
0.795
0.163
0.403
AlI
GTL (LLaMAZ)
LIFT-ICL (GPT-4o)
AlI
P2T (GPT-46)
Fe (GPT-46)
DeLTa (GPT-40) (Ours)
0.116
0.780
0.351
0,529
0.200
0.130
0.351 ()) 12.9%)
TabLLM (TO)
$12
LIFT (GPT:3.5)
0.432
1.167
0.752
1.385
0.589
0.368
0.782
TP-BERTa (RoBERTa)
0.186
0,837
0.528
0.760
0.383
0,280
0.496
512
GTL (LLaMA2)
512
LIPT-ICL (GPT-4o)
(GPT-4o)
512
FeaiLLM (GPT-410
(GPT-44) (Ours)
(1 6.9%)
256
TaHLLM (TO)
256
LIFT (GPT-3.5)
0.324
1.034
0.683
1.294
0.578
0.379
0.715
256
TP-BERI's (ReBERTa)
0.359
0.833
0.557
0.798
0.545
0.368
0.582
(LLaMA2)
256
LIFT-ICI. (GPT:40)
(GPT-4o)
256
FeaLLLM (GPT-4o)
256
DeLTa (GPT-4o) (Ours)
0.234
0.841
0.568
0.799
0.397
0.277
0.519 (U 10.7%)
128
TabLLM (TO)
LIFT (GPT-3.5)
0,290
0.710
0.660
0.430
0.743
128
TP BERTs (RoBERTa)
0.431
0.903
0.597
0.850
0.654
0.390
0.642
128
GTT (LLaMA2)
LIFT-ICL (GPT-40)
128
P21 (GPT-40)
FeaLLLM
128
Del.Ta (GPT-4o) (Ours)
0.247
0.857
0.641
0.855
0.512
0.319
0.572 () (1.0%)
TabLLM (TO)
64
LIFT (GPT-3.5)
0.610
1.050
0.810
1.320
0.700
0.470
0.827
(RoBERTa)
0.727
64
GTL (LLaMAA2
0.663
1.190
0.834
1.189
0.724
0.483
0.847
LIFT-ICL (GPT-46)
1,213
0.710
0.494
0.820
P2T (GPT-40)
0.697
1.355
0.868
1.452
0.745
0.513
0.935
FeatLLMM (GPT-46)
DeLTa (GPT-40) (Ours)
0.382
0.888
0.701
0.919
0.634
0.383
0.651 10.4%)
TabLLM (TO)
LIFT (GPT:3.5)
0.890
1.130
1.010
1.350
0.920
0.670
0.995
TP-BERTs (RoBERTa)
0.814
0.941
0.924
1.008
0.969
0.669
0.887
GTL (LLaMA2)
0.804
1.184
0.904
1.258
0.854
0.624
0.938
16
LIFT-ICL (GPT-4o)
0.815
1.256
1.074
1.172
0.977
0.855
1.025
(GPT-4o)
1.042
16
Featl. L. (GPT-40)
(GPT-4o) (Ours)
(L 4.0%)
28

---

A.8 Full comparison results with conventional baseline methods
We also provide the full results of DeLTa against non LLM-based baseline methods in full-data regimes in Table 12 and Table 13
Table 12:
Comparison with non LLM- based methods on all classification datasets 1. We also calculate the average relative improvement DeLTa against the best baseline method (? 3%).
Methods
BL f
CR
Car
BA f
AD
JA f
Average
KNN
0.7072
0.6429
0.7730
0.8903
0.8186
0.5168
0.7248
CART
0.7871
0.6829
0.7253
0,8900
0.8392
0.5642
0.7481
MLP
0.7171
0.7043
Rores
0.8061
0.7371
0.7944
0.8915
0.8589
0.6588
0.7911 1
XGBoost
0.7605
0.7171
0.6102
0.9051
0,8700
0.7140
0,7630
Catboost
0.7757
0,7543
0.6036
0.9080
0.8726
0.7217
0.7727
ResNet
0.6464
0.7200
0.6332
0.9020
0.8512
0.7134
0,7444
FT-Trasffrmer
0.7871
0.7229
0.7862
0.9030
0,8590
0.7272
SAINT
0.7490
0.7486
0.7582
0.9051
0.8560
0,7050
0,7870
TabPPN (Version 2)
0.7286
0.9095
0
0.7417
ModernNCA
0.8099
0.7286
0.7368
0.9046
0.8702
0.7248
0,7958
DeLTa (Ours)
0.8289
0.7829
0.8355
0.9080
0.8677
0.7048
0.8213 (1 3%)
Table 13: Comparison with non LLM- methods on all regression datasets 4. We also calculate the average relative improvement of DeLTa against the best baseline method (1 2.4%).
Methods
CP4
CRR
CA
HO4
FR4
D1.
Average
KNN
0.2347
1.0955
0.5893
0.7970
0.4628
0.2135
0.5658
CART
0.2533
0.8196
0.7100
0.8433
0.6462
0.3504
0.6038
MLP
0.1318
0.8244
0.4628
0.5949
0,2072
0,3068
0.4213
RurdomForest
0.8093
0.5057
0.3712
XGBoost
0.1285
0.7846
0.4037
0.6169
0.2291
0.79
0.3835
Catboost
0.1210
0.7779
0.3723
0.5672
0.2024
0.1306
0,3619
ResNet
0.1362
0.8194
0.4434
0.5931
0,2083
0.2143
0.4025
FT TTannsforee
0.1187
0.7622
0.3991
0.5851
0.2067
0,1680
0.3766
SAINT
0.8276
0.2063
0,3916
TabPEN (Version 2)
0.1375
0.7956
0.3440
0.5527
0,2000
0. 1282
0.3547
ModernNCA
0.1242
0.7876
0.3656
0.2027
DeLTa (Ours)
0.1156
0.7804
0.3507
0.5293
0.2000
0.1303
0.351 (L 2.4%)
A.9 Computational efficiency
Table 14: Runtime in seconds of DeLTa and other LLM- based baseline methods for the training and inference phase, conducted on Adult dataset. "# Num" indicates number of training samples. +Theee methods require querying LLMs for each test sample. * These methods do not require querying LLMs inference phase.
Stage
TaiLMM
TT- TEBRRTSS
GTL:
LIFTILL:
FeaLLM
DeLTas (Ours)
Train Inference
177371.37 179.1
153660 14
1319,35
N/A
1231.22
35.20
90149488
NIA
0.14
Train
191.75
N/A
1047.55
Inference
19042
89684.76
5.01
153501 OZ
176726.13
200258.24
0.14
0
29

---

A.10
Ablation results
Let F(x) denote the well-trained ensemble of decision trees (Random Forest), and R the correspond- ing decision tree rule set. DeLTa first performs rule refinement procedure via LLM to derive new rule r", and subsequently applies an error correction mechanism to compute a sample-specific error correction vector based on r". which is then added to F(x) to produce the final prediction. We further conduct ablation study to demonstrate the effectiveness of key components of DeLTa, as illustrated in Table 15 and Table T6. by comparing DeLTa against different groups of variants.
Variant A to B do not use the rule refinement process or the error correction process: (1) variant A, that is identical to the single decision tree (CART); (2) variant B, that is identical to the existing ensemble decision trees F(x) (Random Forest).
Variant C to D incorporate the rule refinement process and utilize the r" to predict labels, while removes the error correction process: (3) variant C, that treats the r" as standalone decision tree by assigning the imput a specific leaf node; (4) variant D, that further appends r" to the existing ensemble of decision trees F(z).
Variant E includes both the rule refinement process and the error correction process, and utilizes the r" to predict both labels and error correction vectors: (5) variant E, that uses r" to predict labels like variant D. and also uses r" to predict enror correction vectors like DeLTa. The final output is the sum the predicted label and the predicted error correction vector.
Variant F removes the rule refinement process, but remains the error correction process: (6) variant F, that replaces the LLM generated r" with E R derived from Random Forest to predict the error correction vector, and then adds it to the existing ensemble of decision trees F(x) as the final prediction.
Table 15: Analysis on the effects of different components of DeLTa on classification tasks, "RF" corresponds to Random Forest.
Variants
Rule Relinement
Need RF's output
Esror Correction
BL
CRT
Car
BA f
AD
JA
Averuge
(CART)
0.787
B (RF)
0.806
0.737
0.794
0.892
0.859
0.659
0.791
(for label)
0.811 0.766 0.818
0.900
0.853
0.677
0.804
(For label)
0.812
0.771 0.819
0.899
0.856
0.671
0.805
(Tor residual and label)
v
0.816
0.779
0.829
0.895
0.858
0.679
0.809
DeLTa (Ours)
v /foor residual)
0.829 0.783
0.836
0.908
0.868
0.705
0.821
Table 16: Analysis on the effects of different components of DeLTa on regression tasks. "RF" corresponds to Random Forest.
Variants
Rule Refinement
Need RF's output
Error Correction
CPJ
CRR 1 CA L H0.
FR1
DI1
Average 1
A (CART)
0.253
0.820
0.710
0.843
0.646
0.350
0.604
(RF)
0.152
0.506
0.669
0.163
0.445
C
(for label)
0.136
0.808
0.402
0.607
0.272
0.139
0.394
D
V (for label)
0.144
0.800
0.387
0.583
0.258
0.138
0.385
V (for residual label)
0.125
0.379
0.375
F
0.218
0.819
0.544
0.11
0.619
0.324
0.556
DeL.Ta (Ours)
residual)
0.130
0.351
In the Table in the original paper, DeLTa w/o "RR" corresponds to the variant F, and DeLTa w/o "EC" corresponds to the variant C. Here, "RR" denotes the gule refinement process, and "EC" denotes the error correction process. The results show that the decision tree rule refinement via LLM is important. Directly using the refined rule r* to predict labels could also enhance the performance (variant C and D). And the етrоr correction strategy could further enhance tabular prediction performance (variant and DeLTa). However, without r", the effect of error correction is limited (variant F). Therefore, the removal of any of the components degrades the performance DeLTa.
30

---

A.11 Hyper parameter sensitivity analysis
We incorporate the sensitivity analysis for the number of LLM querying times for DeLTa in Fig. is reasonable for us to set the querying times to 10 by default.
NAMSE
0.36
Accuracy
0.82
0.35
0.80
0.34
7
8
10
Query times
Query times
Figure 7: The effect of the number of LLM queries on performance averaged over all datasets.
A.12
Additional analysis
Fig- 8 shows the label predictions ours: F(x) + A₂ and ours w/o Ar: F(x). We can observe that two categories of samples are overlapped in some regions. F(z) struggles to distinguish them, but
DeLTa could correct the prediction errors of F(x) to enhance the prediction for such complex data patterns.
Labell
Labell
Label2
Label2
Label2
Ground Truth
(b) Predicted RandomForest
(c) Predicted ours
Figure 8: T-SNE visualization of label prediction of DeLTa w/ and w/o error correction vector Ag (i.e., Random Forest) on BA dataset.
We compute the intra-node sample distance averaged over all leaf nodes partitioned by refined rule r" and random forest rules, respectively. The results in Table 17 show that LLM could generate a better rule, where samples grouped within the same leaf node exhibit greater statistical similarity.
Table 17: Average intra-node distance comparison.
BL
Credit
Car
Bank
AD
JA
Average
RandomForest Rule
2,8520
6.0938
2.6609
5.0186
4.7781
9.9177
5.2202
Refined Rule
1.3551
5.0503
2.5059
4.0078
4.2950
9.4011
4.5525 (L 12.79%)
31

---

We also provide an example of the original decision tree rule set derived from Random Forest in Fig. 9 along with the corresponding refined rule generated by the LLM in Fig. [0] This example is taken from the low-data regime with 16 training samples on the Credit dataset.
Tree rules:
feature
-.. class: 1.0
feature 2 0.05
feature 8 33=x-0022 -0.26
feature 10 <m -1.05
J- class: 0.0
Teature 10 -1.05
class: 1.0
feature 8 -0.26
|- class:
Tree rules:
--- feature 16 <= 0.82
--- feature 15 <= -0.99
|- class: 0.0
feature_ 15 -0.99
1- feature <== 0.20
|- class: 1.0
feature 5 0.20
feature
0,18
feature 10 -1.05
t class: 0.0
feature 10 -1.05
- class: 1.0
feature 18 > 0.18
class: 1.0
feature 16 0.82
< -0.91
|- class:
feature 4 -0.91
class: 0.0
Tree 2 rules:
feature 2 < 0.05
--- class: 1.0
feature 2 0.05
|- class: 0.82
feature
feature 17> 0.82
class: 1.0
Figure 9: An example of three tree rules obtained by the Random Forest.
32

---

To design more effective decision rule based on the given CART tree rules and enhance performance, we should aim to combine the features that appear frequently and are pivotal in making decisions, while also ensuring that the rule structure is concise and balanced.
From the provided rules, features like feature feature 10', 'eatture 16", and feature 17 appear play significant the rules.
decision splits. By analyzing depth and split thresholds, we can refine
Here's un improved rule set in the specified format, incorporating insights from the provided trees:
"pyytonn self.tree = {
"feature":
"threshold": 0.05,
"operator": "<==".
"left": ("id": "leaf 1"),
"right":
"feature": 16,
"threshold": 0.82,
"operraor": "<=",
"left": {
"feature": 10,
"hreshold":-I.06
"operator": "<=",
"left": ("id": "leaf_2"),
"right": {
"feature": 18,
"threshold": 0.18,
"operator": ">
"lefi": ("id": "leaf_3"" 3"),
"right": ("id": "leuf 4"),
1.
"right":
"cature":
"threshold": 0.82,
"operator": "c="
"left": ("id"; "leaf_S")
"right": ("id": "Icaf 6").
).
#A# Explanation:
*Choosing Features *
Feature '2' is a critical deciding factor across multiple trees, with consistent threshold of '0.05'.
Feature '16' was chosen due to its significance in Tree with prominent threshold of 00.82.. - Feaature '10' serves as key deciding factor for certain conditions, particularly effective at distinguishing classes when less than '-105".
Features 17' and '18' have been adapted for further splits the right branch, consistent with their impactful splits in Tree and Tree respectively.
**Structure***
The initial split on 'eature immediately classifies a region into leaf
The decision structure on the right branch further splits using feature 16', guiding to other regions depending on values of feature 10', 'feature_ 18", and 'feature_11 17".
This refined rule set strategically uses key features to balance simplicity and depth, aiming for improved decision-makiie accuracy and compactness in the decision space division.
Figure 10: An example of LLM's response for rule refinement.
33

---

How an LLM refines decision tree rules.
(i) The intuition. Traditional methods like CART use information gain to make a series of locally optimal decisions, allowing a data-driven algorithm to find the best feature splits. This can lead to rules that are brittle or overfit the specific data sample used for that tree. DeLTa, instead, utilizes the LLM to analyze diverse rules from Random Forest and performs a logic-based reasoning task: (a) recognizes patterns across multiple "expert" rules to identify features and decision paths that are consistently important. (b) It abstracts the core logic splits from individual trees, (c) It synthesizes a new, globally coherent rule that represents consensus of the most robust patterns.
(ii) A concrete example of a better rule. As shown in Fig. and Fig. [0] we provide a set of decision tree rules from Random Forest and refined rule by the LLM respectively. This is from the Credit dataset, where each sample (person) is classified as good or bad credit risks according to the set of features. To provide deeper insight into how the refined rule improves upon the original ones, also provide the average SHAP values [70] of features under the Random Forest and our method respectively in Table 18 SHAP assigns each feature importance value (higher is more important) for a particular prediction, thus it serves an unified framework for interpreting predictions,
The results show that, after LLM refinement, the SHAP importance of features such as "credit_ "hirttt_istorr" (feature 16) and "employment" (feature_ 10) are increased, which aligns with the financial expertise. Let us take closer look into Fig. (nnd Fig. [0] for "credit history" (feature 16), this feature appears near the top of the second tree, indicating its potential significance. However, only I out 3 decision trees in the original Random Forest rule explicitly splits on this feature, which may dilute its influence in the ensemble prediction. For "employment" (feature_ 10), although it appears in 2 out of 3 rules, its splits occur late in the tree, leading to low overall contribution in prediction. In contrast, the LLM-refined rule elevates the prominence of these features.
(iii) Why do traditional techniques not capture such rule? More broadly, a single decision tree with information gain has the risk of overfitting. Although Random Forest ensembles the outputs from all tree rules, the inherent relationships and interactions among these rules ignored. As the number of trees increases, analyzing these independent rules gradually becoming more and more difficult, let alone utilizing these rules to partition feature space. To this end, we propose to leverage LLMs to analyze and summarize these rules into a refined rule due to the powerful logical reasoning ability of L.M. This enables the generation of improved partitioning strategies over the feature space, thereby promoting statistical coherence among samples assigned to the same leaf node. To verify it, we compute the intra-node sample distance all leaf nodes partitioned by LLM refined rule r", and observe that the distance of r* is lower than original rule r E R from Random Forest, as illustrated in Fig. 2 and Table 17
Table 18: SHAP values of different features.
Originl
ofer.
GD471
chacking
tubce
10025
Swiigs
0,2211
jb 03215
03139
0,070
Oune
plperyy regginnee
cndl vanry
dhecking Saaw
erplases:
oflt (esee plas
Seeign sodke
banigg
Krngs MIRE
Assessing the inherent quality of LLM-rewritten rules. We assess this in a principled way from three perspectives: (i) SHAP value analysis, (ii) intra-node distance comparison and (iii) the transparency of the LLM's process, where the LLM provides a human-readable rationale for its actions, which allows us to inspect and understand its reasoning process.
34

---

Comparison on additional datasets. We have include additional datasets from the WhyTrees Trees [ benchmark. Specifically, we selected subset of representative datasets that span different scales and feature characteristics, including the large-scale, high-dimensional "Year" dataset, to evaluate DeLTa's scalability and generalizability. The updated results, summarized in Table 19 and Table 20 further demonstrate DeLTa's s strong and consistent performance across more diverse and standardized set of tabular tasks.
Table 19: Additional tabular dataset properties. "#objects" indicates the number of samples in the dataset. "#num. features" indicates the number of numerical features, and "#eat. features" indicates the number of categorical features.
#objects
metric
4970
Acc. NRMSE
GiveMeeoooCrodi
16714
Year
515345
90
Table 20: Comparison with baseline methods on additional datasets. "(LY" and "(T)" indicates correspond to L-M--ased and traditional methods, respectively.
Deras
LFT
13W(1)
CART(T)
14.F())
XGBeon::
Callocd 70
FT ITTaadeeer
D417a S1
Gistl G
07765
01554
23310
3.833
63514
0.753
1850
18258
1855
6491
35

---

A.13 Discussion
The existing L...-based methods for tabular prediction suffer from two key inherent issues: (i) data perspective: existing data scrialization methods lack universal applicability for structured tabular data, and may pose privacy risks through direct textual exposure, and (ii) model perspective: LLM fine-tuning methods struggle with tabular data, and in in-context learning scalability is bottle-necked by input length constraints (suitable for few-shot learning). The proposed DeLTa could well solve the aforementioned challenges: Solving the data issue: Unlike serialization methods that convert each sample into unnatural text formats, decision tree rules are composed of simple comparisons between feature values and thresholds, forming logical, interpretable structures that can be naturally expressed in text without relying semantic columns names. In addition, decision tree rules represent global feature space partitioning rule rather than individual samples, which helps mitigate privacy concerns by avoiding exposure of sample-level information. Solving the model issue: Moreover, the powerful reasoning ability of LLMs can be leveraged to redesign decision tree rules and help trees with aggregating their decisions, rather than directly using LLMs generate label predictions. Notably, DeLTa avoids serializing tabular data into natural language format, and does not require additional domain-spedde specific expertise or semantic information, such as explicit feature names and detailed task background knowledge. Furthermore, DeLTa can be applied in full data learning setting without LLM fine-tuning.
Overall, DeL.a achieves the highest average performance in all settings, including classification and regression, in both full and low-data regimes. In addition, DeL.a is computationally efficient in both training and inference stages. This efficiency stems from key properties: (i) DeL.a utilizes the reason ability of LLM without fine-tuning LLM, (ii) DeLTa avoids querying LLMs to generate predictions for individual samples. Instead, it only needs to query LLM via API to generate one refined decision tree rule for one dataset, enabling significantly more efficient use of LLM resources.
A.14 Limitation
Although the primary focus of this work is tabular data, the flexibility of DeLTa opens up exciting possibilities for extensions. For example, DeLTa can be extended to other domains such as time series, graph and so on. We leave the extensions of DeLTa tabular data serialization, it requires training decision trees and extracting corresponding decision rules. However, single decision tree may be prone to overfitting, potentially resulting in suboptimal rules and limited performance improvements in our framework. To mitigate this issuc, we adopt Random Forest to generate diverse set of decision tree rules. This strategy is not only effective but also easy to implement using standard libraries such as scikit-learn, which conveniently supports both model training and rule extraction.
future work. While DeLTa avoids
Another limitation of DeLTa is that its performance advantage over non-LLM baselines narrows with increasing dataset sizes. Upon deeper analysis, we have found that DeLTa's performance advantage more pronounced when the dataset size is relatively small, compared to non-LLM baselines. Conversely, the performance advantage of data-intensive baseline methods, such as neural networks, tends to increase with larger dataset sizes. This pattern may be attributed to two main factors: (i) Nature of Base Rules: Our method refines the base rules extracted from Random Forest (RF). While RF is effective due to its ensemble nature, its trees are independently trained without joint optimization. As the data volume increases, this independent training may limit RF's ability fully leverage large-scale data. Consequently, the quality of the base rules may plateau with increasing data, which also affects the upper bound of the refined rules. (ii) Comparison to Neural Networks (NNs): In contrast, neural networks are typically data-intensive models that require large amounts of training data perform well and generalize effectively. Their performance tends scale better with larger datasets, as they can learn complex feature interactions.
A.15 Future direction
key direction involves developing more scalable base rule extraction mechanisms. Exploring ensembles of jointly optimized trees or hybrid models that better leverage large-scale data could raise the performance ceiling for refinement. Furthermore, integrating the rule-based paradigm with data-intension architectures-fft for instance, by using neural networks to generate pre-select
36

---

candidate rules, or by designing LLM-driven fine-tuming that adapts to dataset scale presents promising avenue combine the interpretability of rules with the scalability of neural models.
A.16 Broader impacts
DeLTa advances the integration of LLMs with structured tabular data by introducing novel way that leverages logical decision tree rules as intermediaries. By representing the entire dataset through decision tree rules, DeLTa avoids the need for sample-level serialization. This makes it particularly well-suited for structured tabular data with heterogeneous features, in contrast to the unstructured nature of textual data. DeLTa also offers practical benefits in terms of data privacy. It could help mitigate privacy concerns by avoiding exposure of sample-level information, which useful in privacy-critical domains such as healthcare and finance. Furthermore, DeL.a can be applied in full data learning setting without LLM fine-tuning thereby offering cost-efficient and scalable solution for integrating LLMs into tabular learning tasks.
37


Reviews:
[{"review_id": "Nru4vnWepC", "reviewer": "Reviewer_sYQb", "summary": "The paper proposes an approach to improve decision trees with LLMs. Specifically, after fitting the CART decision tree, authors feed the decision rules from every tree into an LLM and prompt it to come up with a \"better\" rule. The LLM refined rule is then merged with CART via a gradient net.", "questions": "See comments above regarding the LLM rule refinement and additional analysis of this step.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n-The paper is well written and relatively easy to follow. The proposed approach is innovative and is novel, particularly the LLM rule refinement and merging via gradient net.\n-Results on real world datasets show improvements of the base CART approach as well as other tabular LLM adaptations.\n\nWeaknesses:\n-I think the proposed approach lacks rigor over what LLM is actually doing. Looking at the prompt in Figure 6 of the appendix, after CART rules, the LLM receives the following prompt: \"Based on the above information, please learn the rules evolving process and help me design a better rule like what cart used for inference to achieve higher performance. Please not just copy, please refine these rules and create a new better one.\" Without seeing any of the training data and how it is partitioned by the CART rules, what is a \"better rule\" here? Furthermore, looking at the example output in Figure 11 the LLM generates explanations like \"Feature `16` was chosen due to its significance in Tree 1 with a prominent threshold of `0.82`.\". Here again without seeing how training data is partitioned by this tree, \"significance\" and \"prominent threshold\" seem like random guesses. I think that a rigorous analysis of this step should be conducted to understand how an LLM can refine tree rules without seeing any data, before this paper can be accepted for publication.", "was_revised": true, "final_justification": "Updating recommendation to accept based on author feedback and other reviews.", "rebuttal": "Response to Weaknesses: Thank you for the comment. We agree that the process of refining rules without direct data access must be clearly defined and validated.\nConceptually, \neven without access to raw training data, LLMs are capable of identifying structural patterns in decision rules. This includes detecting which features appear at root nodes, recognizing feature recurrence across trees, and noting threshold values. These patterns are meaningful and often correlate with model salience.  By semantically analyzing these patterns, LLMs acts as a semantic compression engine—reducing complex, low-level decision rules into a higher-level, interpretable representation. In addition, we would like to clarify our framework and demonstrate its soundness by: (1) Clearly defining the LLM's task and its objective. (2) Providing a rigorous, step-by-step analysis of the example in our paper to show the LLM's reasoning is grounded, not random. (3) Proposing a new, qualitative validation to further bolster our claims.\n\n**(1) The LLM's Task: Logical Synthesis of Model Components.** First, we wish to clarify the LLM's role. The proposed approach does not ask the LLM to analyze the data. Instead, its task is one of logical synthesis: it analyzes a set of the decision rules derived from Random Forest to create a new, improved rule. In this context, a \"better rule,\" as requested in the prompt, is defined by superior structural and statistical properties that can be inferred from the provided rules. Specifically, a better rule is one that: Creates more coherent partitions: It groups samples in a way that is more statistically homogeneous. We provide quantitative evidence of this in our paper (Fig. 2 of original paper and Table 17 of Appendix), where the LLM-refined rule consistently achieves a lower average intra-node distance. \n\n**(2) A Grounded Reasoning Process instead of random guess.** To address your question whether the LLM's explanations are \"random guesses\", we walk through the example in Appendix A.12 to demonstrate that the LLM's reasoning is directly grounded in the structural evidence provided in Fig. 10. The LLM states, \"Feature 16 was chosen due to its significance in Tree 1 with a prominent threshold of 0.82.\" which is not a guess. In the input provided (Fig. 10), feature\\_16 is the root node of Tree 1. Being the first split in a tree is the strongest possible structural indicator of a feature's importance for that particular data subset. However, only 1 out of 3 decision trees in the original Random Forest rule set explicitly splits on this feature, which may dilute its influence in the ensemble prediction. Instead, with our design, the LLM correctly identified this and reported it. \nFor feature\\_10, although it appears in 2 out of 3 rules, its splits occur late in the tree, leading to a low overall contribution in the ensemble of Random Forest prediction. \nIn contrast, the LLM-refined rule elevates the prominence of this feature.\nMoreover, the LLM identifies feature\\_2 as a \"critical deciding factor across multiple trees\". This observation is also grounded in the data: feature\\_2 is the root node of both Tree 0 and Tree 2. The LLM correctly inferred that a feature used as the primary split in two out of three \"expert\" opinions is fundamentally important. The LLM's refined rule (Fig. 11) is a logical synthesis of these grounded observations. \nThis behavior reflects a structured and interpretable reasoning process. It leverages decision tree architecture—particularly the hierarchical importance implied by root-level splits—to produce a coherent explanation and synthesis. This is not randomness; it is an informed analysis.\n\n**(3)  A New Proposal for Rigorous, Qualitative Validation.**\nAs shown in Fig. 10 and Fig. 11 of Appendix, we provide a set of decision tree rules from Random Forest and refined rule by the LLM respectively. This is from the Credit dataset, where each sample (person) is classified as good or bad credit risks according to the set of features. To provide deeper insight into how the refined rule improves upon the original ones, we also provide the average SHAP values [1] of features under the Random Forest and our method respectively as follows. SHAP assigns each feature an importance value (higher is more important) for a particular prediction, thus it serves an unified framework for interpreting predictions. The results show that, after LLM refinement, the SHAP importance of features such as ''credit\\_history'' (feature\\_16) and ''employment'' (feature\\_10) are increased.  These features are well-recognized as critical indicators in credit scoring, and their elevated importance in the refined rule reflects closer alignment with established financial domain expertise.\n\nWe are grateful for your detailed feedback, which has pushed us to more clearly articulate and validate the core mechanism of our work. Based on your review, we will revise the paper to include a dedicated section that formally defines the LLM's task, includes the step-by-step logical analysis of our example, and presents the new SHAP validation. We believe these changes will provide the rigor you have requested and substantially improve our paper.\n\n| Original feature       | Importance | Our feature          | Importance |\n|-----------------------|------------|-----------------------|------------|\n| property\\_magnitude   | 0.1487     | property\\_magnitude   | 0.1055     |\n| other\\_payment\\_plans | 0.0471     | credit\\_history       | 0.0436     |\n| checking\\_status      | 0.0410     | checking\\_status      | 0.0342     |\n| purpose               | 0.0355     | purpose               | 0.0174     |\n| housing               | 0.0326     | employment            | 0.0155     |\n| foreign\\_worker       | 0.0251     | other\\_payment\\_plans | 0.0144     |\n| job                   | 0.0213     | foreign\\_worker       | 0.0132     |\n| credit\\_history       | 0.0189     | housing               | 0.0106     |\n| savings\\_status       | 0.0105     | savings\\_status       | 0.0096     |\n| personal\\_status      | 0.0070     | job                   | 0.0095     |\n\n[1] A Unified Approach to Interpreting Model Predictions. NeurIPS 2017."}, {"review_id": "m1oropvw4I", "reviewer": "Reviewer_tga2", "summary": "The paper addresses key challenges in applying LLMs to tabular data, namely the limitations of data serialization and the difficulties of model fine-tuning. The proposed solution, DeLTa (Decision Tree Enhancer with LLM-derived Rule for Tabular Prediction), uses an LLM not to predict outcomes directly, but to refine a set of decision tree rules. First, a Random Forest model is trained on the data, and its decision rules are extracted. These rules are then fed into an LLM via a carefully constructed prompt, which instructs the LLM to synthesize a new, improved rule. This refined rule is then used to partition the data to train a \"Gradient Net\" that predicts and corrects the errors of the original Random Forest model. The authors demonstrate through extensive experiments that DeLTa achieves state-of-the-art performance on various classification and regression tasks, outperforming both LLM-based and conventional tabular data models. A key advantage of DeLTa is that it avoids direct data serialization, thus preserving privacy, and does not require fine-tuning the LLM, making it computationally efficient.", "questions": "Could you provide a concrete example of a set of initial decision rules from a dataset and the corresponding refined rule generated by the LLM? What specific logical simplifications, generalizations, or corrections does the LLM typically introduce?\n\nHow does the performance of DeLTa vary with the complexity of the base model? For instance, how does it change if the initial rules are derived from a much larger and deeper Random Forest versus a smaller one?\n\nThe paper states the method is \"domain knowledge-agnostic.\"  However, could the LLM be using its own embedded knowledge related to feature names (when not anonymized) to refine the rules? How was this potential confounding factor addressed in the experiments?\n\nHow sensitive is the model's performance to the structure and wording of the prompt used to guide the LLM's rule refinement? \n\nWhat is the failure mode of the LLM in this task? Are there instances where the LLM produces a rule that is syntactically incorrect or logically inferior to the original rules? If so, how are these cases handled?", "limitations": "The paper itself acknowledges and discusses limitations in the appendix.  Based on the paper's content, the primary limitations are:\n\nThe performance of DeLTa is inherently tied to the reasoning capabilities of the underlying LLM. A less advanced LLM might not produce effective rule refinements. \n\nThe process relies on effective prompt engineering to guide the LLM. The quality of the refined rule could be sensitive to the specific prompts used. \n\nWhile the method is more efficient than fine-tuning, it still relies on querying a large, often proprietary, LLM, which can incur costs and latency. \n\nThere is a lack of deep qualitative insight into the \"reasoning\" the LLM applies. It is not fully clear what kinds of logical patterns or external knowledge the LLM uses to improve the rules.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths\nNovel Approach: The core idea of using an LLM to refine logical rules from decision trees, rather than serializing raw data, is highly innovative. It creates a new interface between LLMs and structured data models, moving beyond simple data-to-text conversion. \n\nWell-Written Framework: The paper presents the DeLTa framework in a clear, logical, and well-structured manner. The motivation is well-established, and the methodology is described with sufficient detail, making the novel concepts easy to follow. \n\nStrong Empirical Results: DeLTa shows promising and consistently strong performance across a variety of classification and regression datasets, under both full-data and few-shot settings. It outperforms other LLM-based methods and is competitive with traditional state-of-the-art models like XGBoost and CatBoost. \n\n\nPrivacy-Preserving and Efficient: By operating on decision rules instead of individual data samples, the method avoids exposing potentially sensitive data to the LLM. Furthermore, it does not require computationally expensive LLM fine-tuning, making it a more practical and scalable solution. \n\n\nWeaknesses\n\nLimited Qualitative Analysis: While the paper proves that the refined rules are better (e.g., lower intra-node distance), it doesn't provide enough insight into how they are better.  A qualitative analysis showing concrete examples of original vs. LLM-refined rules would be invaluable to understand the reasoning and \"world knowledge\" the LLM contributes.\n\nNeed for Broader Validation: Although the experiments are extensive, validation on an even wider range of datasets, particularly those with very high dimensionality or different types of feature distributions, would further solidify the claims of the model's generalizability.\n\nDependency on Powerful LLMs: The best results are achieved using GPT-4. The performance is likely sensitive to the quality of the LLM used, and the method's effectiveness might decrease with less capable models. The prompt engineering itself is also a critical, and potentially sensitive, component.", "was_revised": false, "final_justification": "", "rebuttal": "R to W1 \\& Q1: Thanks for your valuable comment. We kindly remind you that we have provided a detailed example in Appendix A.12. As shown in Fig. 10, Fig. 11 of Appendix A.12, we provide a set of decision tree rules from Random Forest and LLM refined rule respectively. This is from the Credit dataset, where each sample (person) is classified as good or bad credit risks according to the set of features. To provide deeper insight into how the refined rule improves upon the original ones, we also provide the average SHAP values [1] of features under the Random Forest and our method respectively as follows. SHAP assigns each feature an importance value (higher is more important) for a particular prediction, thus it serves an unified framework for interpreting predictions. \n\nThe results show that, after LLM refinement, the SHAP importance of features such as ''credit\\_history'' (feature\\_16) and ''employment'' (feature\\_10) are increased, which aligns with the financial expertise. Let us take a closer look into Fig. 10, Fig. 11 of Appendix, for ''credit\\_history'' (feature\\_16), this feature appears near the top of the second tree, indicating its potential significance. However, only 1 out of 3 decision trees in the original Random Forest rule set explicitly splits on this feature, which may dilute its influence in the ensemble prediction. \nFor ''employment'' (feature\\_10), although it appears in 2 out of 3 rules, its splits occur late in the tree, leading to a low overall contribution in the ensemble prediction. \nIn contrast, the LLM-refined rule elevates the prominence of these features.\n\nMore broadly, although Random Forest ensembles the outputs from all rules, the inherent relationships and interactions among these rules are ignored. As the number of trees increases, analyzing these independent rules is gradually becoming more and more difficult, let alone utilizing these rules to partition feature space. To this end, we propose to leverage LLMs to analyze and summarize these rules into a refined rule due to the powerful logical reasoning ability of LLM. Although the features in our method are anonymized, the LLM is still able to infer and capture latent patterns and relationships among the original Random Forest rules. This enables the generation of improved partitioning strategies over the feature space, thereby promoting statistical coherence among samples assigned to the same leaf node. This is also demonstrated by Fig. 2 of main paper and Table 17 of Appendix.\n|Original feature|Importance|Our feature|Importance|\n|-|-|-|-|\n|property\\_magnitude|0.1487|property\\_magnitude|0.1055|\n|other\\_payment\\_plans|0.0471|credit\\_history|0.0436|\n|checking\\_status|0.0410|checking\\_status|0.0342|\n|purpose|0.0355|purpose|0.0174|\n|housing|0.0326|employment|0.0155|\n|foreign\\_worker|0.0251|other\\_payment\\_plans|0.0144|\n|job|0.0213|foreign\\_worker|0.0132|\n|credit\\_history|0.0189|housing|0.0106|\n|savings\\_status|0.0105|savings\\_status|0.0096|\n|personal\\_status|0.0070|job|0.0095|\n\n[1] A Unified Approach to Interpreting Model Predictions. NeurIPS 2017.\n\nR to W2: Following your suggestion, we further add experiments to verify the generalizability of our DeLTa on the large-scale dataset with high dimensionality. The dataset details and experimental results are summarized  as follows. The results show that DeLTa also achieves well performance on the high dimensional dataset. We will include more datasets in the revision.\n|Dataset|\\#objects|\\#num. features|\\#cat. features|metric|\\#classes|\n|-|-|-|-|-|-|\n|Year$\\downarrow$|515345|90|0|NRMSE|--|\n\n|LIFT|TPBERTa|KNN|CART|MLP|RandomForest|XGBoost|CatBoost|FTTransformer|DeLTa|\n|-|-|-|-|-|-|-|-|-|-|\n|1.054|0.8258|1.0459|0.9374|0.8134|0.8547|0.8376|0.8157|0.809|**0.8015**|\n\nR to W3's first part: Thanks for raising the important point about the dependencies of DeLTa on powerful LLMs. As we show in Appendix A.6, DeLTa with Qwen3-32B (open-source) already achieves performance comparable to that with GPT-4o. To further strengthen this point, we have conducted new experiments with an even more accessible Qwen3-8B (open-source). The results are summarized below. The results clearly show that DeLTa maintains its strong and stable performance across all three LLMs, even with the much smaller Qwen3-8B. This is because we task the LLM with a structured, logic-based synthesis of decision rules, not a complex, open-ended creative task. The core patterns in the provided rules are distinct enough to be identified by a range of capable models.\n|\\# Num|Various LLM backbones|BL|CR|Car|BA|AD|JA|Average|\n|-|-|-|-|-|-|-|-|-|\n|All|DeLTa (Qwen3-8B)|0.817|0.766|0.816|0.906|0.861|0.692|0.810|\n|All|DeLTa (Qwen3-32B)|0.821|0.766|0.832|0.907|0.867|0.703|0.816|\n|All|DeLTa (GPT-4o) (Ours)|0.829|0.783|0.836|0.908|0.868|0.705|0.822|\n|128|DeLTa (Qwen3-8B)|0.744|0.679|0.680|0.751|0.771|0.456|0.680|\n|128|DeLTa (Qwen3-32B)|0.744|0.687|0.679|0.749|0.778|0.456|0.682|\n|128|DeLTa (GPT-4o) (Ours)|0.746|0.683|0.722|0.749|0.774|0.457|0.688|\n|64|DeLTa (Qwen3-8B)|0.734|0.667|0.594|0.725|0.758|0.460|0.656|\n|64|DeLTa (Qwen3-32B)|0.736|0.657|0.567|0.729|0.759|0.454|0.650|\n|64|DeLTa (GPT-4o) (Ours)|0.732|0.663|0.615|0.733|0.758|0.457|0.660|\n\nR to W3's second part \\& Q4: Thanks for your comment. The design of our prompt is shown in Eq. 3 of main paper and Fig. 6 of Appendix. To assess the sensitivity of DeLTa’s performance to the structure and wording of the prompt, we conduct an ablation study on its design. We consider the following variants: Variant 1 directly removes the whole part of meta information $p_{meta}$. Variant 2 softly rephrases the first sentence of the requirement part $p_{requirement}$ to: ''Based on the above information, please understand these rules and provide a refined rule.'' Variant 3 removes the second sentence of requirement $p_{requirement}$, which originally states: ''Please not just copy, please refine ...'' The results conducted on Blood dataset ($\\uparrow$) show that removing the meta information or slightly rewording the requirement does not lead to significant performance degradation, suggesting that the model is robust to moderate variations in prompt phrasing. However, removing the instruction of ''Please not just copy, please refine ...'' leads to a notable performance drop. This indicates that explicitly guiding the LLM to go beyond copying and to discover interactions among Random Forest rules is crucial for the effectiveness of DeLTa.\n|DeLTa|Variant 1|Variant 2|Variant 3|\n|-|-|-|-|\n|0.829|0.825|0.827|0.815|\n\nR to Q2: Following your suggestion, we analyze the performance of DeLTa with different complexity of the Random Forest by increasing the maximum number of leaf nodes allowed in each decision tree (ML: max\\_leaf\\_nodes) and the maximum depth of each individual decision tree (MD: max\\_depth). The following results conducted on Blood dataset ($\\uparrow$) show that moderately increasing model complexity enhances the LLM’s ability to refine the decision rules. However, further increasing the complexity yields limited benefit. Based on these observations, we conclude that DeLTa does not require an overly complex base model. A Random Forest with moderate complexity is sufficient to provide the necessary rule diversity for DeLTa to achieve a desired performance, highlighting a practical and efficient aspect of DeLTa. We will add this analysis into our revision.\n|ML=10 (Ours)|ML=15|ML=20|\n|-|-|-|\n|0.829|0.834|**0.835**|\n|**MD=10 (Ours)**|**MD=20**|**MD=50**|\n|0.829|**0.837**|0.836|\n\nR to Q3: Thanks for your comment. To verify whether LLM can use its own knowledge related to feature names to refine the rules, we further conduct ablation study. We add feature names in the format of ''feature\\_0: name, ...'' to the meta information of prompt (Eq. 3 of main paper, Fig. 6 in Appendix). We observed that the performance difference was marginal, when feature names are anonymized or not anonymized, indicating that the model’s ability to generate effective rules does not significantly rely on embedded domain knowledge from feature names. This supports our claim that the method operates in a domain-agnostic manner. \nNotably, operating on the anonymized features can significantly mitigates privacy concerns when using LLM.\n|Variants|BL $\\uparrow$|CR $\\uparrow$|\n|-|-|-|\n|DeLTa (w/o feature names, Ours)|**0.829**|0.783|\n|DeLTa (w/ feature names)|0.821|**0.786**|\n\nR to Q5: Thank you for this insightful question. Regarding logically inferior rules, we observed a rare failure on the Adult 64-shot dataset where the LLM did not synthesize a novel rule from multiple decision tree rules, but instead replicated the structure of one of the input trees with only minor threshold changes. Our framework is designed with two specific mechanisms to be robust against these occasional suboptimal generations. First, we mitigate the risk of a single poor LLM output by querying the LLM 10 times by default and aggregating the results. \nAs shown in our sensitivity analysis in Appendix A.11 (Fig. 7), this aggregation process demonstrably improves and stabilizes performance, smoothing out the impact of any single underperforming generation. \nMore importantly, the error correction stage of DeLTa acts as a powerful safety net. The LLM's refined rule $r^* $ is not the final predictor; its role is to structure the problem for a Gradient Net that learns to correct the errors of the original Random Forest. The effectiveness is validated by our main ablation study in Table 4. Even in the aforementioned failure case where the initial LLM-generated rule was suboptimal on its own, the final performance of the full DeLTa framework still surpassed the original Random Forest baseline. In summary, while we acknowledge that the LLM is not a perfect synthesizer, DeLTa is not brittle. Through a combination of query aggregation and a powerful error correction mechanism, our framework is designed to be robust and consistently deliver state-of-the-art performance."}, {"review_id": "SJzR1oTWAF", "reviewer": "Reviewer_d19o", "summary": "This paper uses a LLM to generate new rules for decision trees. The paper proposes a novel gradient update strategy to incorporate LLM rules into the decision tree inference.", "questions": "See Weaknesses.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- This paper has impressive performance compared to XGBoost, CatBoost, and TabPFN\n- The method of using LLMs to augment decision trees is quite novel\n\nWeaknesses:\n- What is the intuition behind using LLMs to split v.s. classical methods, such as information gain, XGBoost, and CART. I understand the performance is better, but feel hesitant to accept the paper without understanding why the number is better. Are there specific examples where the LLM can generate a better rule than decision trees? Why do traditional techniques, like information gain, not capture such a rule?\n- Using an LLM to generate rules sounds a lot more compute intensive than XGBoost or CART. What is the performance overhead of using the LLM?\n- The datasets used in the paper is not very comparable. Why not use a standardized benchmark like WhyTrees or TabZilla? This makes the state-of-the-art argument a bit hard to accept.\n- The writing could be improved. I had trouble reading through Section 4.2.", "was_revised": true, "final_justification": "The authors have addressed the major concerns I had during the rebuttal period. Accordingly, I changed my score from 3 Weak Reject to 5 Accept.", "rebuttal": "R to W1: Thanks for your valuable comment. \n\n**(i) The intuition.**\nTraditional methods like CART use information gain to make a series of locally optimal decisions, allowing a data-driven algorithm to find the best feature splits. This can lead to rules that are brittle or overfit to the specific data sample used for that tree. DeLTa, instead, utilizes the LLM to analyze diverse rules from a Random Forest and performs a logic-based reasoning task: (a) It recognizes patterns across multiple ''expert'' rules to identify features and decision paths that are consistently important. (b) It abstracts the core logic splits from individual trees. (c) It synthesizes a new, globally coherent rule that represents a consensus of the most robust patterns. \n\n**(ii) A concrete example of a better rule.**\nAs shown in Fig. 10 and Fig. 11 of Appendix, we provide a set of decision tree rules from Random Forest and refined rule by the LLM respectively. This is from the Credit dataset, where each sample (person) is classified as good or bad credit risks according to the set of features. To provide deeper insight into how the refined rule improves upon the original ones, we also provide the average SHAP values [1] of features under the Random Forest and our method respectively as follows. SHAP assigns each feature an importance value (higher is more important) for a particular prediction, thus it serves an unified framework for interpreting predictions. \n\nThe results show that, after LLM refinement, the SHAP importance of features such as ''credit\\_history'' (feature\\_16) and ''employment'' (feature\\_10) are increased, which aligns with the financial expertise. Let us take a closer look into Fig. 10 and Fig. 11 of Appendix, for ''credit\\_history'' (feature\\_16), this feature appears near the top of the second tree, indicating its potential significance. However, only 1 out of 3 decision trees in the original Random Forest rule set explicitly splits on this feature, which may dilute its influence in the ensemble prediction. \nFor ''employment'' (feature\\_10), although it appears in 2 out of 3 rules, its splits occur late in the tree, leading to a low overall contribution in prediction. \nIn contrast, the LLM-refined rule elevates the prominence of these features.\n\n**(iii) Why do traditional techniques not capture such a rule?**\nMore broadly, a single decision tree with information gain has the risk of overfitting. Although Random Forest ensembles the outputs from all tree rules, the inherent relationships and interactions among these rules are ignored. As the number of trees increases, analyzing these independent rules is gradually becoming more and more difficult, let alone utilizing these rules to partition feature space. To this end, we propose to leverage LLMs to analyze and summarize these rules into a refined rule due to the powerful logical reasoning ability of LLM. This enables the generation of improved partitioning strategies over the feature space, thereby promoting statistical coherence among samples assigned to the same leaf node. To verify it, we compute the intra-node sample distance over all leaf nodes partitioned by LLM refined rule $r^* $, and observe that the distance of $r^*$ is lower than original rule $r\\in \\mathcal{R}$ from Random Forest, as illustrated in Fig. 2 of original paper and Table 17 of Appendix. \n|Original feature|Importance|Our feature|Importance|\n|-|-|-|-|\n|property\\_magnitude|0.1487|property\\_magnitude|0.1055|\n|other\\_payment\\_plans|0.0471|credit\\_history|0.0436|\n|checking\\_status|0.0410|checking\\_status|0.0342|\n|purpose|0.0355|purpose|0.0174|\n|housing|0.0326|employment|0.0155|\n|foreign\\_worker|0.0251|other\\_payment\\_plans|0.0144|\n|job|0.0213|foreign\\_worker|0.0132|\n|credit\\_history|0.0189|housing|0.0106|\n|savings\\_status|0.0105|savings\\_status|0.0096|\n|personal\\_status|0.0070|job|0.0095|\n\n[1] A Unified Approach to Interpreting Model Predictions. NeurIPS 2017.\n\nR to W2: Following your suggestion, in addition to the computational cost comparison between DeLTa and LLM-based baselines reported in Table 3 of original paper, we now add a comparison with non-LLM baselines such as XGBoost and CART. The comparison is summarized below, where we provide runtime in seconds of different methods for the training and inference phase, conducted on Adult dataset. ''\\# Num'' indicates number of training samples. ''T'' and ''L'' correspond to traditional and LLM-based methods, respectively. It can be observed that, although DeLTa’s training time is longer than that of traditional tree-based models such as CART, XGBoost, and CatBoost, our DeLTa’s training cost is lower than neural methods designed specifically for tabular data, such as FT-Transformer and MNCA, and substantially lower than LLM-based baselines. The reason is that DeLTa only queries LLM via API to generate one refined rule for one dataset, without the need for per-sample prediction or fine-tuning LLM. Furthermore, at the testing stage,  DeLTa makes decision by combining the trees and the Gradient Net, without querying LLM. Therefore, we can see that DeLTa exhibits similar inference costs to these tree-based methods and neural network-based methods, and incurs substantially lower cost compared to most of LLM-based baselines. To sum up, designed in this way, DeLTa enables significantly more efficient use of LLM resources, whose training and inference workflow are summarized in Algorithm 1 in the Appendix A.3.\n| \\# Num | Stage | KNN (T)| CART (T)| RandomForest (T)| XGBoost (T)| CatBoost (T)| FT-Transformer (T)| MNCA (T)| TabLLM (L)| LIFT (L)| TP-BERTa (L)| FeatLLM  (L)| DeLTa (Ours) (L)|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n| All | Train | 0.90 | 0.10 | 0.28| 0.85 | 12.58  | 277.26| 92.55  | 177371.37 | 153689.14 | 1319.35 | 1231.22  | 35.20  |\n| All | Inference | 2.11| 0.07 | 0.08 | 0.08| 0.17  | 0.92 | 0.72| 179.21 | 90149.48 | 1.58 | 0.14 | 0.09 |\n\nR to W3: Thank you for raising this important point. We kindly remind you that our primary objective was to evaluate DeLTa's effectiveness in the context of LLM-based tabular prediction, and to ensure fair comparison with existing methods in this category. Accordingly, the majority of our classification datasets were selected based on their widespread use in recent LLM-based tabular learning studies such as TabLLM and FeatLLM, both of which primarily focus on classification tasks. To extend our evaluation beyond classification, we also incorporated regression datasets from the TALENT benchmark [2][3], which has recently gained attention for assessing tabular regression performance. In response to your suggestion, we have updated our experimental setup to include additional datasets from the WhyTrees benchmark. Specifically, we selected a subset of representative datasets that span different scales and feature characteristics, including the large-scale, high-dimensional \"Year\" dataset (as also recommended by Reviewer tga2), to evaluate DeLTa's scalability and generalizability. Due to time constraints during the rebuttal phase, we were only able to include a subset of WhyTrees datasets. However, we plan to expand this evaluation to a broader range of WhyTrees datasets in the final version. The updated results, summarized below, further demonstrate DeLTa's strong and consistent performance across a more diverse and standardized set of tabular tasks.\n|Dataset|\\#objects|\\#num. features|\\#cat. features|metric|\\#classes|\n|-|-|-|-|-|-|\n|rl|4970| 5| 7 | Acc. | 2 |\n| GiveMeSomeCredit | 16714     | 10              | 0               | Acc.   | 2         |\n| Year             | 515345    | 90              | 0               | NRMSE  | --        |\n\n|Dataset| TabLLM (L) | FeatLLM (L) | LIFT (L) | TP-BERTa (L) | KNN (T) | CART (T) | MLP (T) | RandomForest (T) | XGBoost (T) | CatBoost (T) | FT-Transformer (T) | DeLTa (L) |\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n| rl $\\uparrow$| 0.7716| 0.7243| 0.6349| 0.7394| 0.6258| 0.6308   | 0.6630  | 0.7022           | 0.7726      | 0.7746       | 0.7072             | **0.7897**    |\n| GiveMeSomeCredit $\\uparrow$ | 0.7765     | 0.7137      | 0.6534   | 0.7810       | 0.6063  | 0.7514   | 0.7469  | 0.7768           | 0.7789      | 0.7792       | 0.7610             | **0.7816**    |\n| Year $\\downarrow$           | -  | - | 1.0540   | 0.8258       | 1.0459  | 0.9374   | 0.8134  | 0.8547           | 0.8376      | 0.8157       | 0.8090             | **0.8015**    |\n\n[2] A Closer Look at Deep Learning on Tabular Data. 2024.\n\n[3] TALENT: A Tabular Analytics and Learning Toolbox. 2024.\n\nR to W4: Thanks for the comment. We will carefully revised Section 4.2 to make it more clear. The core purpose of Section 4.2 is to introduce a calibration method for the original decision tree ensemble, $F(x)$. This is done by computing a sample-specific error correction vector, $\\Delta_x $, to steer the prediction in the direction of error reduction. The process is as follows. Objective: We aim to adjust $F(x)$ by moving it toward the negative gradient of the loss function, which we define as the \"error-reducing\" direction. Partitioning: We use the refined rule $r^* $ (from Section 4.1) to partition the \"Gradient Set\"—a collection of training features and their corresponding negative gradients—into disjoint leaf nodes. This groups samples with similar structures. Learning: For each leaf node, we learn a specific mapping function $\\phi_l$ that models the relationship between sample features and their negative gradients (Eq. 4). Inference and Calibration: For any new input $x$, we use $r^* $ to assign it to a leaf node. The corresponding mapping function $\\phi_l$ then predicts the error correction vector $\\Delta_x $ (Eq. 5). The final calibrated prediction is the sum of the original prediction and this correction: $F(x)+\\Delta_x $ (Eq. 6)."}, {"review_id": "no5JZDcvWn", "reviewer": "Reviewer_WTPj", "summary": "This paper introduces DeLTa (Decision Tree Enhancer with LLM-derived Rule for Tabular Prediction), a novel approach that integrates Large Language Models (LLMs) with decision trees to improve predictions on tabular data. According to authors, DeLTa addresses three key issues in prior LLM-based methods for tabular data:\n1) lack of universal applicability and privacy risks from data serialization usually used in current LLM-based tabular approaches.\n2) challenges of LLM fine-tuning\n3) scalability limitations in-context learning\n\nThis approach seems to be using logical decision tree rules as intermediaries and it leverages the reasoning ability of LLMs to generate improved rules from existing decision tree rule sets without requiring LLM fine-tuning.", "questions": "- The rule r* may contain threshold or other information depending on the feature types. Does this leak privacy in some form?\n- Apart from the experimental results, which are showing good performance, how can we measure if the LLM rewriting rules (or correcting them) is performing well? In a way, this is also a bit of a zero-shot task for the LLM itself.\n- How is η set? what's the interpretation of it in the context of DeLta?\n- Is it fair to say that the method always assumes that the LLM will always find a r* leading to lower loss but this is an assumption?", "limitations": "Maybe clarify:\n- Privacy leakage beyond samples\n- Discuss when LLM fails to improve rules", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The authors seem to deliver on their promise to avoid serialization, fine tuning and scalability. The main advantage of the paper seems to rely on improved rules from existing decision tree rule sets and using these rules to calibrate predictions. The method achieves state-of-the-art performance on diverse tabular benchmarks in both full-data and low-data regimes.\n\nVery interesting approach to combine 2 steps to refine tree rules and training a gradient net to correct errors.\n\nMy main comment is that this seems more of decision tree enhancer system that introduces advantages natural to LLMs rather than a LLM-based solution for tabular data problems. And that's fine, the way the authors use an LLM to synthesize a refined rule r* from multiple tree rules, and then apply that to structure a residual correction layer, reminds me of boosting. Would you say DeLTa is a boosting-inspired framework but the LLM refines instead of being a data-driven iterative fitting?\n \nMaybe also citing TabNet (it's one of the most well-known deep learning models for tabular data with attention and interpretability), or TabTransformer (pioneered using transformers on column embeddings).\n\nTypos:\n- throughough\n- an domain", "was_revised": true, "final_justification": "That's great. Thanks", "rebuttal": "R to W1: Thanks for your thoughtful comment. You are right that DeLTa shares some conceptual resemblance with boosting. \nTraditional Boosting would train a new, simple weak learner (e.g., a shallow decision tree) on the residuals, allowing a data-driven algorithm to find the best feature splits. DeLTa, instead, utilizes the LLM to analyze diverse rules and then synthesize a feature space partitioning rule, $r^*$. This rule then structures the error-correction stage. The LLM's role is not a greedy search on data but a logical synthesis of existing rules. \nWhile DeLTa undoubtedly benefits decision trees, its essence is to explore a novel use of LLMs for tabular data, which fundamentally differs from existing LLM-based tabular prediction paradigms.\n\nMost current LLM-based approaches typically first serialize tabular data into natural language descriptions, and then tune LLMs or directly infer on these serialized data. There are two key limitations: (1) it is challenging to design such suitable template due to the modality gap between unstructured text and structured tabular data. And it may also pose privacy risks through direct textual exposure, and (2) LLM fine-tuning methods struggle with tabular data, and in-context learning scalability is bottle-necked by input length constraints (suitable for few-shot learning).\n\nIn contrast, DeLTa integrates LLM into tabular data via logical decision tree rules as intermediaries: the LLM is not used to predict sample-level outputs directly, but instead to generate a refined decision rule by understanding multiple tree rules.\nThis rule-level interface has several distinct advantages: it (a) avoids serialize tabular data into natural language format, (b) preserves data privacy and does not require domain-specific information, and (c) can be applied in full data learning setting without LLM-finetuning.\nIn essence, while the error-correction mechanism is boosting-inspired, the core novelty is using an LLM for logical rule refinement, establishing a new and more robust way to bridge the gap between LLMs and structured data.\n\nR to W2: Thanks for your comment. We will cite these papers in our main manuscript.\n\nR to W3: Thanks for pointing this out. We will carefully revised the corresponding parts of manuscript to make it more clear.\n\nR to Q1: Thanks for your comment. As discussed in Appendix A.4, including the prompt example in Fig. 6,  although we need to submit the rules to LLM, we do not need to upload additional domain-specific semantic information when designing prompt, such as explicit feature names or detailed task background knowledge. While the decision tree rules may contain feature threshold, all features fed into LLM are anonymized. Moreover, decision tree rules represent global feature space partitioning rule rather than individual samples. That is to say, the LLM operates only on abstracted, symbolic representations that reflect model logic rather than raw data. Moving beyond some LLM-based tabular learning methods that expose the serialized tabular samples containing raw feature values to LLMs, ours significantly mitigate privacy concerns.\n\nR to Q2:  Thanks for your comment. We agree that evaluating the quality of the rule rewriting process itself, independent of final task performance, is crucial for understanding why our method is effective. We assess this in a principled way from three perspectives: (i) SHAP value analysis, (ii) intra-node distance comparison and (iii) the transparency of the LLM's process.\n\n**(i) SHAP value analysis.** As shown in Fig. 10 and Fig. 11 of Appendix, we provide a set of decision tree rules from Random Forest and refined rule by the LLM respectively. This is from the Credit dataset, where each sample (person) is classified as good or bad credit risks according to the set of features. To provide deeper insight into how the refined rule improves upon the original ones, we also provide the average SHAP values [1] of features under the Random Forest and our method respectively as follows. SHAP assigns each feature an importance value (higher is more important) for a particular prediction, thus it serves an unified framework for interpreting predictions. \n\nThe results show that, after LLM refinement, the SHAP importance of features such as ''credit\\_history'' (feature\\_16) and ''employment'' (feature\\_10) are increased, which aligns with the financial expertise. Let us take a closer look into Fig. 10 and Fig. 11 of Appendix, for ''credit\\_history'' (feature\\_16), this feature appears near the top of the second tree, indicating its potential significance. However, only 1 out of 3 decision trees in the original Random Forest rule set explicitly splits on this feature, which may dilute its influence in the ensemble of Random Forest prediction. \nFor ''employment'' (feature\\_10), although it appears in 2 out of 3 rules, its splits occur late in the tree, leading to a low overall contribution in the ensemble of Random Forest prediction. \nIn contrast, the LLM-refined rule elevates the prominence of these features.\n|Original feature|Importance|Our feature|Importance|\n|-|-|-|-|\n|property\\_magnitude|0.1487|property\\_magnitude|0.1055|\n|other\\_payment\\_plans|0.0471|credit\\_history|0.0436|\n|checking\\_status|0.0410|checking\\_status|0.0342|\n|purpose|0.0355|purpose|0.0174|\n|housing|0.0326|employment|0.0155|\n|foreign\\_worker|0.0251|other\\_payment\\_plans|0.0144|\n|job|0.0213|foreign\\_worker|0.0132|\n|credit\\_history|0.0189|housing|0.0106|\n|savings\\_status|0.0105|savings\\_status|0.0096|\n|personal\\_status|0.0070|job|0.0095|\n\n[1] A Unified Approach to Interpreting Model Predictions. NeurIPS 2017.\n\n**(ii) Intra-node distance comparison.** Although Random Forest ensembles the outputs from all tree rules, the inherent relationships and interactions among these rules are ignored. As the number of trees increases, analyzing these independent rules is gradually becoming more and more difficult, let alone utilizing these rules to partition feature space. To this end, we propose to leverage LLMs to analyze and summarize these rules into a refined rule due to the powerful logical reasoning ability of LLM. This enables the generation of improved partitioning strategies over the feature space, thereby promoting statistical coherence among samples assigned to the same leaf node. \nTo verify it, we compute the intra-node sample distance over all leaf nodes partitioned by LLM refined rule $r^* $, and observe that the distance of $r^*$ is lower than original rule $r\\in \\mathcal{R}$ from Random Forest, as illustrated in Fig. 2 of original paper and Table 17 of Appendix.\n\n**(iii) Transparency of the LLM's Process: The Self-Explanation.** Finally, our method is not a complete black box. The LLM provides a human-readable rationale for its actions, which allows us to inspect and understand its reasoning process. As shown in Fig. 11 of Appendix, the LLM generates a detailed \"Explanation\" for its refined rule. It explicitly breaks down its logic into two parts: \"Choosing Features and Thresholds\" and \"Structure\". It articulates why feature\\_2 was chosen (as a \"critical deciding factor across multiple trees\" ) and how it constructed the new tree (the \"initial split on feature\\_2 immediately classifies a region into leaf 1\" ). \n\nIn summary, we evaluate the LLM’s rule-rewriting performance through a multi-faceted approach. Together, these methods provide a robust and principled way to measure the success of this core component of our work. We will revise our manuscript to clarify it.\n\nR to Q3: $\\eta$ is set to 0.2 by default. In Eq. 6 of original paper, we need to move prediction produced by the Random Forest $F(x)$ in the direction of negative gradient by one step, and $\\eta\\in \\mathbb{R}_+$ is the hyperparameter controlling the step size. In this context, $\\eta$ plays a similar role to the learning rate in gradient descent, and it determines the strength of the correction. We found that $\\eta=0.2$ works robustly.\n\nR to Q4: Thanks for this insightful observation. You are correct that DeLTa relies on the LLM's ability to generate a useful rule $r^* $. However, we do not assume this process is deterministically perfect. Notably, DeLTa is designed with two specific mechanisms that make it highly robust to occasional, suboptimal generations from the LLM. First, we mitigate the risk of a single poor LLM output by querying the LLM 10 times by default and aggregating the results, as discussed in Appendix (Line 558). This helps to reduce variance and smooth out occasional underperforming generations. We have indeed observed rare cases (e.g., on the Adult 64-shot setting) where the LLM failed to synthesize a novel rule and instead largely replicated an input tree. Our aggregation strategy is designed to minimize the impact of such infrequent events. Second, and more importantly, the error correction (EC) stage of DeLTa acts as a powerful safety net, ensuring a strong final performance even if the initial LLM-generated rule is imperfect. The LLM refined rule $r^* $ is not the final predictor. Its primary role is to structure the problem for Gradient net, which then learns to correct the errors of the original Random Forest . Even a modestly effective $r^* $ can provide a useful partitioning of the problem space for this error correction mechanism to build upon. The effectiveness is demonstrated in ablation study in Table 4 . The results show that the DeLTa significantly outperforms the variant that uses only the LLM-refined rule for prediction (w/o ''EC''). This proves that the EC stage provides a substantial performance uplift and ensures the final model is robust. Therefore, while the LLM refinement is a key driver, DeLTa is not brittle. Through a combination of robust aggregation and powerful error correction, DeLTa is designed to consistently achieve performance improvements, even when the effectiveness of a single LLM invocation cannot be strictly guaranteed."}]

Author rebuttal:

Meta-review: This work proposes an LLM-based method to refine decision tree ensembles during training for tabular data prediction tasks. By providing an LLM with a textual description of trained trees from a random forest ensemble, the LLM is prompted to recommend changes to the tree structure that could improve performance. As opposed to most tree-training algorithms which are greedy and optimize each split locally, this refinement step can take a wider view on the structure of the tree and which features are important. Since LLMs are only used for refinement, not inference, the method retains the fast inference properties of tree-based models.

The reviewers are aligned that the paper is high enough quality to be accepted. Through the reviews, some issues were brought up including: that the method seems to lack rigor; intuitive understanding of why LLMs can refine decision trees is lacking; qualitative analysis is lacking; that LLMs could cause efficiency overhead; and that validation could be done on larger datasets.

Through the rebuttals the authors addressed most of these concerns. It was explained that LLMs can use semantic information across trees rather than greedy local feature splitting, and that at inference time no LLMs are needed which helps efficiency. The authors provided some qualitative examples of the refinement in action, and presented more baselines, datasets, and ablations as requested by reviewers. Although this does not cover a more rigorous understanding of how and why the method works, the intuitive explanations have been improved.

Hence, I am recommending to accept this submission. The authors must revise their work to take account of the reviewer’s suggestions and the discussions.
Decision: Accept (spotlight)