Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning
Authors: Jaehyun Nam
Kyuyoung Kim
Seunghyuk Oh
Jihoon Tack
Jaehyung Kim
Jinwoo Shin
Venue: NeurIPS 2024 poster (2024)
Keywords: Tabular Learning
Large Language Models
Primary area: natural_language_processing

Abstract: In tabular prediction tasks, tree-based models combined with automated feature engineering methods often outperform deep learning approaches that rely on learned representations. While these feature engineering techniques are effective, they typically depend on a pre-defined search space and primarily use validation scores for feature selection, thereby missing valuable insights from previous experiments.
To address these limitations, we propose a novel tabular learning framework that utilizes large language models (LLMs), termed Optimizing Column feature generator with decision Tree reasoning (OCTree). Our key idea is to leverage the reasoning capabilities of LLMs to identify effective feature generation rules without manually specifying the search space and provide language-based reasoning information highlighting past experiments as feedback for iterative rule improvements. We use decision trees to convey this reasoning information, as they can be easily represented in natural language, effectively providing knowledge from prior experiments (i.e., the impact of the generated features on performance) to the LLMs. Our empirical results demonstrate that OCTree consistently enhances the performance of various prediction models across diverse benchmarks, outperforming competing automated feature engineering methods. Code is available at https://github.com/jaehyun513/OCTree.

Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning
Jaehyun Nam"l Kyuyoung Kim"l, Seunghyuk Oh'
Jihoon Tack², Jachyung Jinwoo Shin
*KAIST, 2 Yonsei University
(jaehyun. nam, kykim, inwwos) wwwikkkaaaa...... kr
Abstract
In tabular prediction tasks, tree-based models combined with automated feature en- gineering methods often outperform deep learning approaches that rely on learned representations. While these feature engineering techniques are effective, they typically depend on pre-defined search space and primarily use validation scores for feature selection, thereby missing valuable insights from previous experiments. To address these limitations, we propose novel tabular learning framework that utilizes large language models (LLMs), termed Optimizing Column feature gen- erator with decision Tree reasoning (OCTree). Our key idea is to leverage the reasoning capabilities of LLMs to identify effective feature generation rules with- out manually specifying the search space and provide language- based reasoning information highlighting past experiments as feedback for iterative rule improve- ments. We decision trees to convey this reasoning information, they be easily represented in natural language. effectively providing knowledge from prior experiments (i.e., the impact of the generated features on performance) the L.Mss. Our empirical results demonstrate that OCTree consistently enhances the performance of various prediction models across diverse benchmarks, outper- forming competing automated feature engineering methods. Code is available at attps://githir /gtb/btttthb.../.gg com/ jaehyu5533OOCTres
Introduction
Learning useful representations from raw data is key to the success of deep learning algorithms, and their effectiveness has been demonstrated across multiple domains, e.g., vision [1, 2, 3, 4] and language [5, 6]. However, in the tabular domain, deep learning approaches
often perceived less effective [7, 8, 9, 10). For instance, tree-based approaches utilizing raw column features of tabular data (11, 12] often outperform deep learning models in tabular prediction tasks such classification and regression [13, 14, 15]. As a result, practitioners commonly resort to using tree-based methods coupled with manual feature engineering, such computing the product of two column features [16].
Generating suitable column features, even with domain knowledge, can challenging and costly. For instance, manual validation to identify useful features is infeasible due to the exponentially many possible combinations to explore [17]. To address this issue, existing feature engineering methods [17, 18. 19] use additional filtering schemes [20, 21, 22] evaluate and select useful features automatically. While these approaches reduce manual effort and improve feature quality, they still present several challenges. First, practitioners often rely on manually defined search spaces to generate candidate features due to the inherent ambiguity what constitutes informative features [17, 23). However, this still requires substantial computation for validating candidate features,
'Equal contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Zep
Prepare for the rule optimization
Example Prompt
Prompt
Suggested Feature
Make for optinizatinn
Your dejective predict whather the sutject has the disnase.
Condider follow attributee:
Suggest column
descripiion
LLM
Whether smoking reuted discase
[Column namies]
Suggested feature (ue. whether or not smoking)
(Column named]
Give rafe for generating the suggested feature.
Use LLM as black-box rule optimizer
Create column based on the rule from Step
Prompt
Suggested
Original
Genseata rule
Twwer
Farigue
Breathing
Smokk
Example Prompt Trajecttry*
LLM
Has fever and breothing
Difficutt
Yes
Linat a amocher
2ep3
Evaluate the generated column and extract reasoning
Updated Data
1. Train the prediction model
Tree- based reasoning
Trajectory*
Faver
Fatigue
Breathing
Smoke
Save the result
(Scors, Reasoningg
begaa:: "In Taakestie dioaser
Yes
No
Difccut
Yes
(Soome Reanoning)
You
Fire
(Score, Reasoning!
SHpp4
Repeat steps 1-3 a fixed number of times, then select the rule with the best validation score
Figure 1: Overview of OCTree. (Step 0) Prompt the LLM to propose a name for the new column. (Step 1) Generate rule by prompting the LLM with feedback on previously generated rules and relevant information for reasoning about the data. (Step 2) Generate a new column feature based on the proposed rule. (Step 3) Train a prediction model on the new data and compute the validation score and tree-based reasoning, provided as feedback for iterative improvements. (Step 4) Repeat steps 1-3 fixed number of times and select the rule with the best validation score.
particularly as the number of features and the complexity the search space grow. Furthermore, they neglect more effective experimental designs, relying solely on validation scores to select good features, despite the value of past experiment data for improving selection.
Motivated by this, we propose to approach this problem from a novel perspective: optimization to discover effective generation rules, leveraging the language understanding and reasoning capabilities of large language models (LLMs). Recent research has demonstrated that LLMs can optimize various dindiifferet problems using prompts that describe the optimization task in natural language [24, 25, 26]. This suggests the potential for LLMs to automatically generate and iteratively refine feature generators without the need for manually specifying the rule space. For example, the reasoning capabilities of LLMs allow incorporating feedback on their previous outputs into the process for iterative refinement. Moreover, linguistic contexts, such as column names (e.g., 'Gender' and 'Age") and categorical values (e.g.. 'Female' and 'Male'), could be naturally integrated into the optimization [27, 28, 29, 30J, which is difficult, if not impossible, with conventional methods.
Contributions. In this work, we leverage LLMs to generate novel column features for tabular prediction tasks, proposing Optimizing Column feature generator with decision Tree reasoning (OCTree), generic framework for automated feature generation using LLMss. Figure illustrates an overview of our framework. Our approach begins by prompting an LLM to propose a name for a novel column feature based the task description, such as 'Trading Vo Volume' for stock price prediction. This initial suggestion guides the LLM exploring and refining the corresponding feature values. Then, we further leverage the reasoning capability of LLMs to produce good rule that generates values for the newly introduced column feature based on the existing ones. Specifically, starting from an initial rule ro. we let the LLM iteratively improve the current rule T3 by using extracted reasonings do, .i....,.. de and validation scores 5g, . St attained by the prediction model as feedback. Here, d, denotes the language description of decision tree fitted to the entire dataset, including the new feature generated by rı. Specifically, we propose using the decision tree reasoning to provide the LLM with effective knowledge from the past experiments, i.e., the prediction model trained with the generated features, providing learned knowledge about the entire dataset. This procedure is iterated for fixed number of times, after which select the rule with the highest validation score.
We assess the effectiveness of OCTree on wide range of real-world datasets (e.g., stock price and patient mortality prediction) from various sources, including recent Kaggle competitions. Our

---

experimental results demonstrate that OCTree consistently enhances the performance of various prediction models, including gradient-boosted boosted decision trees [11] and deep neural networks [31, 321, for both classification and regression tasks. We also assess OCTree on datasets where language descriptions are unavailable, i.e, all feature values and column names are anonymized during preprocessing. Even on these datasets, OCTree reduces relative prediction errors by an average of 5.0% compared to the best baseline model, i.e., XGBoost on the 19 classification tasks benchmarked by Grinsztajn et al. [13]. Here, we use the Llama 2 7B model fine-tuned on high-quality dialogue data to enhance its ability to understand and generate contextually relevant, coherent rules. We also show that OCTree outperforms recent automatic feature engineering methods, including CAAFE [19] and OpenFE [17], often using our 7B LLM, even when these methods are combined with significantly more advanced models like GPT-4. Lastly, we demonstrate that features generated for one type of model (e.g. a simpler model like XGBoost) can enhance the performance of other model types (e.g., more complex models like neural networks). This illustrates a potential approach to scaling the method for larger, more complex models.
2 Related work
Tabular learning with LLMss Recent developments in L.MSs have encouraged investigation into their applications to tabular prediction tasks. Dinh et al. [27] and Hegselmann et al. [28] fine-tune GPT-3 [33] and TO [34], respectively, by serializing tabular data into natural language. Nam et al. [35] utilizes unlabeled data expressed in natural language for few-shot semi-supervised tabular classification tasks via prompting LLMs. More recently, Yan et al. [36] introduced a tabular-specific tokenization method to pre-train a single language model capable of performing well across multiple tabular datasets. Instead of using LLMs as prediction models, we explore whether they can effectively generate useful column features for tabular prediction tasks. Specifically, we propose enhancing various prediction models by using LLMs as optimizers to generate novel column features.
LLMs as optimizers. Various prompting techniques have demonstrated the use of LLMs for solving optimization problems. This is achieved by describing optimization problems natural language and instructing LLMs to iteratively generate new solutions based on previously found solutions and their evaluation. In particular, Yang et al. [25] uses LLMs to optimize linear regression, the traveling salesman problem, and prompt optimization (i.e., refining instructions to improve LLM outputs). Building on these insights, we leverage LLMs to optimize feature generators for tabular prediction tasks. Unlike prior work, we incorporate decision tree reasoning feedback, providing the model with learned knowledge about the dataset in natural language for more effective optimization.
Automated feature engineering, Automated feature engineering involves generating features from raw data without human effort to improve the performance on prediction tasks [19]. Various methods have been developed for this purpose [37, 38, 39], including iterative feature subsampling with beam search to select informative features [23] and feature boosting and pruning algorithms for efficient and accurate filtering [17]. More recently, Hollmann et al. [19] introduced a context-aware feature engineering approach that leverages LLMs to generate semantically meaningful features based on the description of given task. Unlike previous approaches, our method leverages the optimization and reasoning capabilities of LLMS to discover effective feature generation rules without the need for manually defining search space. Furthermore, while methods such CAAFE [19] rely on language-bart based context, our approach is applicable to both context-aware and context-agnostic settings, enabling it to handle broader range of prediction tasks.
3 Optimizing feature generator with decision tree reasoning
In this section, we introduce a framework for automated tabular feature engincering by leveraging the language understanding and reasoning capabilities of LLMs. In a nutshell, our approach utilizes LLMs as optimizers to propose and refine the rules for generating column features. Specifically, we iteratively improve the rules by guiding the LLMMs using (1) the validation performance of previously proposed rules and (2) decision tree-based reasoning derived from the training data as inputs, enabling more effective optimization. We begin by framing the feature engineering problem as an optimization of the rules for generating column features (Section 3.1) and then introduce the core framework, termed Optimizing Column feature generator with decision Tree reasoning (OCTree), designed to solve this optimization task (Section 3.2).
3

---

Problem setup. Formally, the goal of tabular prediction tasks is to train a prediction model f X V. where X' is the imput space and X is an M M-dimensional column feature with corresponding column names C = {c,....; Car) J. For example, 'x1: Female' and "az: 36' are values for the columns 'c1: Gender' and "02: Age', respectively. In classification tasks, y E y {0, 1 represents the label space with K classes, while in regression tasks, y E E C RR We denote by Orarget the name of the column corresponding to the labely.
3.1 Tabular feature generation as rule generator optimization
We frame tabular feature engineering as the optimization of feature-generattn rules, where the rules define a mapping from the original set of features to a new feature. Our objective to generate a one-dimensional column feature, X", by optimizing the rule r : X' - X". Specifically, we aim to create a novel column feature that improves the performance of the prediction model when trained with the new feature, f: X X" + V. Our optimization problem can be formalized as follows:
min L @ r) subject to f = arg min Ly (Derrin # r):
where g is the rule generator (i.e., LLM M in our case), := (Derain)) is the rule generated based denotes the dataset augmented with
on the training dataset Perains and D ( == fx, 2 r(x)).))) yi yi); ) the new column feature generated. L, is the objective function for the given prediction task, such as mean absolute error for regression tasks, evaluated using the model f. In summary, we optimize the rule r to achieve the best validation score measured on Dval @ with the model f" trained to minimize the loss on Dersin @ r.
However, such bi-level optimization is often non-differentiable or computationally demanding, as involves computing gradients through the optimization of f*, Morcover, the rule itself may involve non-differentiable operations, such as logical conjunctions between categorical features (e.g.- 'Is a Smoker = Has Fever A Has Difficulty Breathing' as Figure 1). One approach to addressing the issue is to use black-box optimization methods, such as evolutionary strategies [40] or reinforcement learning [41]. However, these methods also have limitations, including the need for manually defined search space, which is complex, as well as the potentially suboptimal use of valuable feedback from previously proposed solutions that could enhance the optimization process.
To address this issue, we propose leveraging LLMs as optimizers for the tabular feature engineering problem. Our approach involves iteratively proposing and refining rules by prompting LLMs with a trajectory of feedback, which includes the history of previously proposed rules, the validation scores, and the associated reasoning information. Optimization using LLMS [25] has proven to be an effective tool, particularly for black-box optimization problems, which we show can also be effective in tabular feature engineering. Furthermore, LLMs can leverage the semantics of column and feature values for better optimization and provide the flexibility to operate without a pre-defined search space.
3.2 Generating column features with OCTree
We now present the core algorithm. First, we prompt the LLM to propose a name of a new column, such 'Smoking Status' in Figure 1, and the corresponding rule based the task description. We then compute the validation score of the prediction model and extract decision tree reasoning from the training dataset, initializing the optimization trajectory. This trajectory provides feedback to the LLM, with the decision free reasoning component, which effectively conveys the knowledge of past experiments and captures the quality of previously suggested features. As the optimization process continues, the trajectory is updated, enabling the L.LM to refine and enhance the rule iteratively.
Column name generation. OCTree begins by generating name of new column feature, Cneus through the LLM M. This is done by prompting M with the prompt Peo1 (see Ap- pendix A. which asks for a new column name that would be useful for predicting the target: Cnew = MPcoo (C, Ctarget).. Leveraging its language understanding capabilities, the LLM is able to generate semantically coherent and relevant column names. For instance, it might suggest using trading volume as a new feature for predicting stock prices.
Initialize optimization and extract decision tree reasoning. OCTree then generates an initial rule To for deriving a new column feature from the original set of columns C. This is done by prompting the LLM M with the prompt Pinit (see Appendix A.2) to propose a rule for predicting Caew with

---

CI.....CM Cari o (Miiii (C, Czex)). The score so for the initial rule is then evaluated with the prediction model f so Ly. (Dval ro). Additionally, decision tree reasoning dg is extracted using CART [42] fitted to the training dataset:
dg = CART ① ro).
CART a binary decision tree that recursively splits the data based on criteria such as Gini impurity to predict the outcome. We employ CART for two main reasons: (i) tree-based models, often ensembles of simple decision trees like CART, outperform deep learning on many tabular prediction tasks, and (ii) CART is easily interpretable and can be expressed in natural language. For example, as illustrated in Figure CART can be expressed using simple iflelse syntax. Intuitively, the decision tree reasoning extracted by CART provides valuable insights learned from the entire training dataset. explicitly highlights the columns that are considered more significant (as nodes in the tree) and the corresponding values (as thresholds of the nodes) used for prediction.
Optimization with decision tree reasoning. To optimize the rule, we describe the task in natural language and provide the trajectory T. = {(s:, d, ri)):=o which includes the history of previously proposed rules, the corresponding scores, and associated reasoning information. We then generate a new rule T using the LLM M with the prompt Pgen (see Appendix A.3), which asks M to propose a new rule that is not present in Tr and that would improve on the scores from the previous iterations: T++1 (Pgea (Te, C, Crarge=)). )). The elements in the optimization trajectory are ordered by the scores, as LLMs tend to generate suggestions that appear later in the list [25, 43]. Afterward, we append the new score 8 = (Daai @ re+1), the decision tree reasoning d CART(Derain @ Tr+ ). and the rule r to the trajectory: T =
Tr U {(+++ r )] The optimization proceeds for fixed number of iterations, with the best rule selected based on the highest validation score.
Generating multiple features. The optimiza- tion process can be repeated to generate multiple useful features. For example, after generating the column 'Smoking Status', an additional column PPhysial Activity Level' can be generated based on the original features and the newly created 'Smoking Status'. Formally, we first generate new column X' (X'), where Topt is the optimized rule for generating the new feature X'. This results in an augmented input space, Xneer = X' & X'. Using the updated dataset Dawv C xee x V. OCTree iteratively generates additional column features. This process continues, introducing new features in sequence until the validation score no longer improves.
Original data X
X
B
Generate
w/ Topt
High
No
Hard
Yes
Male
Casy
Femele
High
Yes
Fasy
Figure 2: Generation of multiple features. The optimization process is repeated to generate multi- ple column features in sequence.
Experiments
In this section, we evaluate the effectiveness of OCTree across a range of tabular classification and regression tasks using diverse datasets. Our findings demonstrate that OCTree consistently improves the performance of various types of prediction models (Section 4.1 and Section 4.2). Furthermore, ablation studies confirm the value of the proposed decision tree reasoning in the optimization and demonstrate that features generated using one type of prediction model can effectively transfer to others, suggesting an approach to scaling the framework to more complex models (Section 4.3).
Datasets. First, we select real-world datasets with language descriptions of the column features from diverse sources, including the Discase, Academic, Enefit, and Tesla Stock datasets, recently released on Kaggle, and the Clinical Trial dataset from the US National Library of Medicine. These prediction tasks are highly practical and relevant to domains such as healtheare (e.g., diagnostics), academia (e.g., student dropout prediction), and finance (e.g., stock price forecasting). In addition, to evaluate OCTree on datasets without language descriptions of the columns, we include the 19 classification datasets benchmarked by Grinsztajn et al. [13]. When selecting datasets, it is important to consider the heterogeneous nature of tabular data [44] and ensure coverage of both categorical and numerical features, well as both classification and regression tasks. We conduct experiments on this diverse selection of datasets to evaluate OCTree's 's general applicability across various types of tabular data. Further details on the datasets are provided in Appendix B.
5

---

Table 1: Performance improvement by OCTree datasets with language descriptions. We report test error rates (%) for three classification tasks (*) and mean absolute error (x10-3) for two regression tasks (t). The lowest errors are highlighted in bold. Values in parentheses indicate the relative error rate reduction from the baseline. We report the mean error and standard deviation across three random splits, except for two regression tasks (time series tabular data), which are split by time index. N/A indicates that the method is not applicable, as HyperFast is a classification model.
Method
LLM
Tesla
Enefit
Discuse"
Clinical
Academic
XGBoosi [11]
Baseline OCTree OCTree
6.61
8.00 8.00 (0.0%)
28.09
46.27=50
14,15
Llama 2
5.56 (15.9%)
26, 26,19 (6.8%)
45.07=441 (2.6%)
14.11=0s (0.3%)
GPT-4o
5.48 (17.1%)
7.82 (2.3%) %)
25.72 (8.4%)
43.75 (5.4%)
13.74 (2,9%)
MLP [31]
Baseline OCTree
7.41
33,53
38.10+32
41.77++17
14,41 +08
Llama
5.23 (29.4%)
29.99 (10.6%)
32.861x7 (13.7%)
39.80 29.8 (4.7%)
14.26 (1.0%)
OCTree
GPT-4o
5.01 (32.4%)
21.68 (35.3%)
30.95 (18.8%)
39.25 (6.0%)
14.22=oo (1.3%)
HyperFast [32]
Baseline OCTree
N/A
N/A
28.57=100
43.64
14.67=07
Llama 2
N/A
N/A
28. 10 (1.6%)
41.45
14,49 (1.2%)
OCTree
GPT-4o
N/A
N/A
27.14 (5.0%%)
42.00=15 (3.8%)
14.49=os (1.2%)
Baselines. To validate our method, we evaluate it across three types of prediction models. We first consider XGBoost [11], a highly competitive tree-based model known for its effectiveness in the tabular domain. Second, we apply our method to multilayer perceptron (MLP; Gorishniy et al. [3I], the base architecture for deep learning models. Lastly, we show that OCTree improves the performance of HyperFast [32], a recently introduced model designed for fast classification of tabular data. Implementation details, including hyperparameter search space, are provided in Appendix C.
Common setup. For all datasets, 60% of the data is used for training, 20% for validation, and 20% for testing. Following Gorishniy et al. [31], we use learned embeddings for categorical features when training MLPs, while HyperFast handles categorical features automatically. For all experiments, we use CART with a maximum depth of extract decision tree reasoning, provided to the rule generating LLM in the prompt. Unless noted otherwise, we use the Llama 2 model [45] at the 7B scale, fine-tuned on UltraChat [46], a dialogue dataset that has been used to develop strong chat models such as UltraLMM [46]. Our findings indicate that open models, even at moderate scales, can be highly effective, particularly when equipped with strong chat capabilities. Further comparisons across different types of LLMs are provided in Section 4.2.
4.1 Main results: Context-aware feature engineering
Datasets with language descriptions. We first experiment with datasets where language descriptions of the columns and categorical feature values are available. In these cases, the L.M generates a logical rule in natural language, which is then converted into Python code for use in our experiments. For this task, we utilize both GPT-4o and our Llama 2 model, as the Llama model can be limited by its relatively short context length on some datasets. See Appendix A.4 for the prompt used.
As shown in Table I, OCTree consistently enhances the performance of various baseline models. For instance, when generating the column 'Trading Volume' for the Tesla Stock dataset using Ilama 2 for XGBoost, the relative error is reduced by 15.9%. OCTree is compatible with arbitrary LLMs, allowing more advanced models to enhance the quality of generated features further. Specifically, with GPT-4o, one of the latest LLMs from OpenAI, our method achieves a relative error reduction of 17.1% on the same dataset for XGBoost. We provide results on additional datasets in Table 13.
Comparison with CAAFE. CAAFE [19] also introduces a feature engineering approach that utilizes LLMs to construct features based on the linguistic context. However, it is important to note that CAAFE requires explicit language descriptions
the features, which limits its applicability when such information is unavailable. For example, feature names and values are often obfuscated for confidentiality, a common practice in the financial and medical domains. In contrast, our method can be effectively applied to datasets without linguistic descriptions, as demonstrated in Table 3.

---

Table 2: Applicability and comparison of automated feature engineering methods. We report the mean error (%) and standard deviation across the six datasets with language descriptions used in Tables and 13. The lowest error is highlighted in bold. Values in parentheses indicate the relative error rate reduction from the baseline model (i.e., XGBoost [11J, while N/I indicates no gain.
Applicability
Comparison
Method
w/o descriptions
w/ descriptions
LLM
Avg. Err. (%)
Baseline
25.87
AutoFeat [23]
25.7612.1 (0.4%)
OpenFE [17]
/
26.44 ( N/1
CAAFE [19]
x
GPT-4o
25.43 (1.7%)
Llama 2
25 (2.9%) 24.53 (5.2%)
OCTree (Ours)
GPT-4o
Thus, to compare CAAFE with OCTree, we evaluate both methods on datasets with contextual information, particularly all of the six classification datasets used in Tables 1 and 13. The results in Table 2 show that our method significantly outperforms CAAFE with GPT-4o, even when using our custom Llama 2 model fine-tuned on open dialogue data. Also, conventional feature engineering methods, such as OpenFE [17], often struggle to generate meaningful features, particularly due to the difficulty of applying arithmetic operations to categorical features. A key distinction between CAAFE and OCTree is that our approach generates more semantically meaningful column names, which serve as a basis for creating high-quality features. Leveraging the LLM's s reasoning and in-context learning capabilities, we guide the model in effectively navigating the feature space to generate coherent, relevant rules, using a history of feedback candidate features and decision tree reasoning to enhance its understanding of the data. In contrast, CAAFE primarily relies on language understanding to suggest simple combinations of existing feature. Moreover, CAAFE tends to adopt a greedy approach, evaluating the validation score for a candidate feature only once and discarding it if no improvement is observed. We provide further results with case studies in Appendix E.
4.2 Main result: Context-agnostic feature engineering
Datasets without language descriptions. In practice, datasets do not always include clear language descriptions of the prediction task. For example, feature names and values in financial datasets are often obfuscated with arbitrary symbols to protect confidentiality [47]. OCTree adapts easily such datasets without language descriptions and can generate features using various arithmetic rules. As shown in our ablations in Section 4.3, this due to the more effective use of LLMs' optimization capabilities with decision tree reasoning that enhances the model's 's understanding of the data.
To evaluate datasets without language descriptions, we apply ordinal encoding to categorical features and normalize all features using min-max scaler, transforming the original numeric values. We also use non-descriptive column names, such as C ('x1', 'x2', ..., 'x5') } for dataset with M columns. To initialize the feedback trajectory, we create an initial rule that is the product of the two columns with the highest importance weights computed using an XGBoost model, e.g., I6 = 22 x 23-- As shown in Table 3, our framework enhances baseline models even in the absence of language descriptions, achieving an average error reduction of approximately 5.0% for both the XGBoost classifier and MLP. For HyperFast, OCTree also improves the test error on 16 out of 19 datasets.
Analysis of experiments with various open LLMs. To evaluate how our method performs with various types of LLMs, we assess the performance of sev- eral open LLMS as rule generators. As shown in Table 4, while all models yield improvements over the baseline, we find our own model (i.e., Llama 2 fine-tuned on UltraChat) be particularly effective. This shows that our framework can be effectively implemented even with open models at a moderate scale, especially those with sufficiently strong chat capabilities. We suspect that this improvement stems from the enhanced ability of these models to understand and generate contextually relevant and
Table 4: OCTree with Llama 2 variants. We report the average test error rates (%) and stan- dard deviations across three random seeds on the 19 datasets without language descriptions.
Method
LLM
Size
Avg. Err.
XGBoost OCTree
16.53
Llama 2 Chat
7B
16.322n.1
OCTree
Code Llama
7B
15.83+n2
OCTree
Ours
7B
15.71
7

---

Table 3: Performance improvement by OCTree on datasets without language descriptions. We report test error rates (%) on the 19 classification tasks from Grinsztajn et al. [13]. The lowest error is in bold. Values in parentheses indicate the relative error rate reduction from the baseline, while N/I
indicates no gain. We report the mean error and standard deviation across the three random splits.
XGBoust
MLP
HyperfFast
Datuset
Baseline
OCTree (Ours)
Baseline
OCTree (Ours)
Baseline
OCTree (Ours)
electricity
8.32
6.651m1 (20.1%) %)
15.64
14.82
5.2%)
15.25
14.70aus
3.6%)
rl
23.61
19.32.aa (18.2%)
32.03
28.30
(11.6%)
33.77
33.50
0.8%)
compass
22.91
18.
(17.6%)
210
26.78:a1
2.3%)
25.74
24.91
3.2%)
covertype
9.10a
7.961u0
(12.5%)
8.73ga4
8,25 ( 5.5%)
9.86211
9.21 ( 6.6%)
phoneme
10.89
10.15
6,8%)
12,06
10,98 ( 9.8%)
10.55=07
10.57=00 ( N/I
kddCup09
19.86
19.07
4.0%)
24.30
21 ( 0.0%)
25.75
24.46
5.0%)
pol
1.69+a2
1.62+02
4.0%)
1.37 +a.
+.3 7.3%)
1,70=04
1.55+02
8,8%)
Magic
a4
13.75
3.5%)
14.60
14.50 ( 0.7%)
14,95
14.34.us
4.1%)
california
9,45
9,13
3.4%)
11,91
11.37
4.5%)
11.75
11.02
6.2%)
house_ 16H
11.66295
11.32:92
3.0%)
13.07
12.54:00
4.1%)
12.77
12.29
3.8%)
eye_ movements
35,06+07
34.17+20
2.6%)
40.03+12
39,86+19
0,4%)
41.33+15
40.29+17(
2.5%)
road-safety
21.14
20.65+ta11
2.3%)
22.17
21.8+
1.4%)
24.54
24.07=044
1.9%)
kdd ipduis la
10.89
10.69
18%%
13.13
11.72
(1..7%
16.15
13,55
(16.1%)
NiniBooNE
5,48+aa2
5.42+a1
1.2%)
9.69+a3
7.35+02
(24.1%)
6,61+04
6.54
1.1%)
credit
22.02.103
21.781103
1.1%)
24.43
23.23
( 4.9%)
25.06.11
24.30
3.0%)
Higgs
27,95
27,91+02
0.1%)
29.43+04
28.80
2.1%)
30.04:02
29.73:05
1.0%)
jannis
20.61291
20.6426.1 ( N/T
22,28
22.51
N/I
24.29
23.65
2.6%)
wine
19.11+33
19.18+39 ( N/I
21.53+31
21.59+L4
N/1 )
19.18+27
19.31 ( N/I
mank-mark
20.09100
2 N/I
21.11
21.09 0.1%)
21.25
21.66105 ( N/I
Table 5: Comparison with automatic feature engineering methods. We report the mean error (%
and standard deviation across the 22 datasets used in Tables and 3. The lowest error is highlighted in bold, and the second lowest is underlined. Values in parentheses indicate the relative error rate reduction from the baseline model. OCTree refers to our method integrated with other approaches.
Prediction model
Baseline
AutoFeat
OpenFE
OCTree (Ours)
OCTree (Ours)
XGBoust
18.30+u.
18.244o. (1.3%)
17.79 (2.8%)
17.45+0u (4.6%)
16.85 -03 (7.9%)
MLP
20.88+uu1
20.60 (1.3%)
20 (3.6%)
19.91 +04 (4.6%)
19.41 +os (7.0%)
coherent rules, resulting in better optimization outcomes. Specifically, as illustrated by the examples in Appendix F, our model navigates a broader space of features more effectively than the base Llama 2 chat model and demonstrates the ability to utilize built-in Python functions such as 'abs()'. Code Llama also demonstrates strong performance due to its training on code data that includes a variety of arithmetic rules, which is especially useful for generating rules for datasets lacking language descriptions. Notably, the model can leverage range of NumPy operations (e.g., 'np.sin') ) allowing it to produce more mathematically complex features. This suggests that LLMs trained on high-qualig, code and dialogue data could serve as even more effective rule generators within our framework.
4.3 Ablations and analysis
Integrating with other automated feature engineering methods. OCTree is complementary to existing feature engineering methods, allowing for integration in several natural ways. One simple approach is to first apply OCTree to generate features, followed by the use of other methods to further refine and augment the feature set. In Table 5, we first compare the standalone performance of OCTree, demonstrating that it outperforms competing state-of-the-an automated feature engineering methods (i.e., AutoFeat [23] and OpenFE [17]) for both XGBoost and MLP models. Moreover, combining OCTree with OpenFE (denoted as OCTreel in Table 5) further boosts performance, achieving a 7.9% reduction in relative error for XGBoost.
Ablation study on the propased components. As shown in Table 6, our framework consists of two essential components: (i) generating new column features (denoted as Gen. Feat. in Table 6), and (ii) providing explicit decision tree reasoning as feedback (denoted DT reasoning in Table 6) during the optimization process. First, note that the rules are sufficiently well-optimized even without an explicit decision tree provided as feedback; the LLM improves performance based solely on score
8

---

Table 6: Ablation study of the proposed decision tree reasoning. We report the mean error (%)
and standard deviation across three random splits on two datasets with language descriptions (*) and two datasets without language descriptions ()). The lowest error is highlighted in bold. Values in parentheses indicate the relative error rate reduction from the baseline model.
Gen. Feat.
DT Reasoning
Disease"
Clinical"
electricity!
kddCup09
28.09279
46.27
8.32
19.86
2.. (1.7%)
45.61+41 (1.4%)
6.89 (17.2%)
19,47=16 (2.0%)
26.19 (6.8% (
45.07=411 (2.6%)
6.65 (20.1 %) 19.07 (4.0%)
Table Performance improvement through feature transfer. We optimize the feature generation rule using XGBoost and transfer the generated features to improve MLP and HyperFast (OCTreetrana). We report the test error rates (%) and standard deviation across three random seeds for two datasets with language descriptions (*) and two datasets without ().. The lowest error is in bold, with values in parentheses indicating the relative error rate reduction from the baseline model. N/I denotes cases where no improvement observed.
MLP
HyperFast
Dataset
Baseline
OCTree (Ours)
Baseline
OCTreenas (Ours)
Discase"
38.10
35.24=444
(7.5%)
28.57
27.62 (5.8%)
Clinical*
41.77+122
42.32+23 N/I )
43.64
42.76 (2.0%)
electricity!
15.64
15.03=03
(3.9%)
15.37
14.88:02 (3.2%)
kddCup09
24.30
23.47=005 (3.4%)
25.62107
25.22 (1.6%)
feedback. However, providing the decision tree feedback to the LLM can lead to even better performance. We believe that decision tree reasoning, which highlights important columns and their threshold values, enables the LLM to understand the data better, resulting in the generation of more contextually relevant and useful rules. Moreover, decision trees can be easily represented in natural language using if-else syntax, effectively conveying the information about the data to the LLM.
Transferring generated rules other prediction models. While we optimize feature generation rules to improve the performance of a specific prediction model, the generated features can also be utilized in other models achieve similar improvements. For example, it would be more efficient to generate features using XGBoost, which typically trains and evaluates faster than larger deep neural networks, and then apply these features to more complex models. To assess whether such feature transfer is feasible within our framework, we first optimize the column generation rules using XGBoost and then train MLP and HyperFast models with the generated features. As shown in Table 7, these features significantly improve the performance of both models, demonstrating the effectiveness of this approach, especially when only limited computational resources are available.
Evaluating the validity of generated features. The rule generator LLM is asked to recommend a new column feature that is not already present in the dataset. Here, we evaluate whether the LLM is capable of generating features that are. in fact, valid and relevant to the target task. We perform this analysis from two perspectives: (i) whether the LLM can identify the most relevant column feature when provided with multiple candidate columns, and (ii) whether using real-world data, when available, for the suggested column leads to improved performance of prediction models.
Our method is based on the assumption that sufficiently capable LLMSs can understand the relationship between the target task and column features, enabling them to generate new, relevant features for the task. To evaluate this assumption, we first examine whether the LLMs can identify the features that are more relevant to the prediction task. For this experiment, we begin by removing two existing features from a dataset and then prompt the LLM to rank the two features according to their importance for the target task. Specifically, we remove 'Cholesterol Level' and 'Cough' from the original Disease dataset and then ask the LLM to identify the attribute more relevant to predicting whether a patient has a disease. Both the GPT-40 and Llama 2 models indicate that 'Cough' more important than "Cooessteool Level'. As shown in Table 8, XGBoost achieves a lower error rate when trained with the 'Cough' feature compared to when trained with 'Cholesterol Level', which consistent with the LLMs' assessment that the former is more relevant to the task.

---

Table 8: LLM identifies important features. We port the mean error (%) and standard deviation across three random splits on the Disease dataset. Both GPT- 4o and Llama 2 identify the cough feature as more im- portant, consistent with the accuracy seen in XGBoost models trained with and without these features.
56
Accuracy (5)
52
Column feature
Model
Cough
Cholesterol
XGBoost
50
w/o Age w/ Age Figure 3: Imputing features with real data, Age. We report the mean ac- curacy (%) across three random splits on the Clinical dataset using XGBoost.
X
34.76:208
/ 33 208
x
30.00+43
/
28.09#79
Building on the LLM's ability identify important features for the target task, we further assess whether the generated columns can be imputed with real-world data to enhance prediction. In this experiment, we use the Clinical Trial dataset, where the LLM introduced the column 'Age' when prompted to suggest a new column. We then incorporated real-world age data from the US National Library of Medicine to augment the original dataset. As shown in Figure 3, training on this augmented data results in notable improvement, demonstrating that the L.L.-generated columns align well with real-world data. conclusion, we recommend that practitioners utilize OCTree to either (i) identify additional column features and collect the corresponding real-world data (ii) optimize feature generation rules in the absence of real-world data.
Analysis of the rule optimization process. We analyze how the output rules evolve throughout the optimization rounds on the electricity dataset. Due to space constraints, we show the first and last five output rules in Appendix H. In the early stages of optimization, the LLM generates diverse range of outputs, indicating an active exploration of potential rules. In contrast, during the later stages, the LLM focuses on refining the solution space around previously identified rules, making only minor adjustments. This again demonstrates that with appropriate guidance through the optimization process, sufficiently capable LLMS can serve highly effective optimizers.
Handling ballucinations. While LLMs may occasionally suggest suboptimal or semantically incoherent rules, our method is designed to address these hallucinations. Specifically, we provide feedback on previously generated rules to guide the LLMs iteratively improving the rule generation. This feedback loop helps the LLMs avoid hallucinations that might lead to low validation scores in subsequent iterations. Empirically, we find that these issues are more common in the early stages when the LLMs explore the rule space more broadly and in less capable models, e.g., those without additional training on dialogue or code generation data.
Conclusion
In this paper, we propose OCTree, generic framework that leverages the power of LLMs (e.g.. reasoning capability) for automatically generating column features for tabular prediction tasks. We evaluate the effectiveness of OCTree across various prediction tasks and demonstrate that our method consistently enhances the performance of diverse prediction models, often significantly more effectively than competing feature engineering methods. As future work, exploring feedback-based alignment methods, such as reinforcement learning from human feedback, to further enhance LLMs as rule generators would be an exciting direction to explore.
Limitation. One potential limitation our work is that evaluating the generated features involves computing the validation scores of the prediction model, which can be time-consuming if the model requires extensive training. However, as demonstrated by the results in Table 7, this issue can be mitigated by first generating features with simpler prediction model and then transferring those features to the target model, reducing the overall runtime and computational requirements.
10

---

Acknowledgements and disclosure of funding
We would like to thank Sihyun Yu, Subin Kim, Changyeon Kim, Daewon Choi, Jinseong Park, and anonymous reviewers for their helpful feedback and discussions. This work was supported by Institute for Information communications Technology Promotion (IITP) grant funded by the Artificial Intelligence Graduate School Program 1022-0-00188 84, Development and Study Al Technologies to
Korea government (MSIT) R---000---11900000 (KAIST); ARR--2022-I12222 inexpensively Conform to Evolving Policy on Ethics), the NIPA (National IT Industry Promotion Agency), through the Ministry of Science and ICT (Hyperscale Al flagship project), and Hyundai Motor Group.
References
[1] Alex Krizhevsky, Πya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[2] Christian Szegedy, Wei Liu, Yangging Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9, 2015.
[3] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729 9729-972299 9038, 2020.
[4] Yonglong Tian, Dilip Krishman, and Phillip Isola. Contrastive multiview coding. In Com- puter Eeiion-ECYY 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Purt XI 16, pages 776-794. Springer, 2020.
[5] Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 2018.
[6] Lajanugen Logeswaran and Honglak Lee. An efficient framework for learning sentence repre- sentations. arXiv preprint arXiv: 02899, 2018.
[7] Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela van der Schaar. Vime: Extending the success of self-and semi-supervised learning to tabular domain. Advances in Neural Information Processing Systems, 33: 100301131000 1003,, 2020.
[8] Xin Huang, Ashish Khetan, Milan Cvitkovie, and Zohar Karnin. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprint 20Xi::0012006675, 2020.
[9] Talip Ucar, Ehsan Hajiramezanali, and Lindsay Edwards. Subtab: Subsetting features of tabular data for self-supervised representation learning. Advances in Neural Information Processing Systems, 34:18853-18866 2021.
[10) Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, and Mahsa Shoaran. Xtab: Cross-table pretraining for tabular transformers. arXiv preprint arXiv:2305, 06090, 2023.
[11] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785-794, 2016.
[12] Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. Advances in neural information processing systems, 31, 2018
[13] Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? Advances Neural Information Processing Systems, 35:507-520. 2022.

---

[14] Kuan- Yu Chen, Ping-Han Chiang, Trompt: Towards better deep neural network for tabular data, arXiv preprint vrXiv:235..18444 2023.
Chou, Ting- Wei Chen, and Tien-Hao Chang.
[15] Duncan McElfresh, Sujay Khandagale, Jonathan Valverde, Ganesh Ramakrishnan, Micah Goldblum, Colin White, et al. When do neural nets outperform boosted trees on tabular data? Advances in Neural Information Processing Systems, 2023.
[16] Valeriia Cherepanova, Roman Levin, Gowthami Somepalli, Jonas Geiping, C Bayan Bruss, Andrew Gordon Wilson, Tom Goldstein, and Micah Goldblum. A performance-drive bench- mark for feature selection in tabular deep learning. Advances in Neural Information Processing Systems, 2023.
[17] Tianping Zhang, Zheyu Aqa Zhang, Zhiyuan Fan, Haoyan Luo, Fengyuan Liu, Qian Liu, Wei Cao, and Li Jian. Openfe: Automated feature generation with expert-level performance. In International Conference on Machine Learning, pages 41 880-419911 PMLR, 2023.
[18] Avinash Amballa, Anmol Mekala, Gayathri Akkinapalli, Manas Madine, Naga Pavana Priya Yarrabolu, and Przemyslaw A Grabowicz. Automated model selection for tabular data, arXiv preprint arXiv:2401.00961, 2024.
[19] Noah Hollmann, Samuel Müller, and Frank Hutter. Large language models for automated data science:
Introducing caafe for context-aware automated feature engineering. Advances in Neural Information Processing Systems, 36, 2024.
[20] Leo Breiman. Random forests. Machine learning, 45:5-32, 2001.
[21] Jerome H Friedman. Greedy function approximation: gradient boosting machine. Annals of statistics, pages 1189-1232, 2001.
[22] Huan Liu and Lei Yu. Toward integrating feature selection algorithms for classification and clustering. IEEE Transactions knowledge and data engineering, 17(4):491-50 2005.
[23] Franziska Horn, Robert Pack, and Michael Rieger. The autofeat python library for automated feature engincering and selection. In Machine Learning and Knowledge Discovery in Databases: International Workshops of ECML PKDD 2019, Würzburg, Germany, September 16-20, 2019, Proceedings, Part I, pages 1 11--20. Springer, 2020.
[24] Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rock- täschel. Prompthreeder: Self-referential self-improvement via prompt evolution. arXiv preprint 16X1:97309.16797. 2023.
[25] Chengrun Yang, Xuezhi Wang. Yifeng Lu, Hanxiao Liu, Quoc Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. arXiv preprint arXiv:2309. 03409, 2023.
[26] Michael R Zhang, Nishkrit Desai, Juhan Bae, Jonathan Lorraine, and Jimmy Ba. Using large language models for hyperparameter optimization. arXiv e-prints, pages arXiv-2311 2023.
[27] Tuan Dinh, Yuchen Zeng, Ruisu Zhang. Ziqian Lin, Michael Gira, Shashank Rajput, Jy-yong Sohn, Dimitris Papailiopoulos, and Kangwook Lee. Lift: Language-interfaced fine-tuning for non-language machine learning tasks. Advances in Neural Information Processing Systems, 35: 11763-11784, 2022.
[28] Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. Tabllm: Few-shot classification of tabular data with large language models. In International Conference on Artificial Intelligence and Statistics, pages 5549-5581. PMLR, 2023.
[29] Hariharan Manikandan, Yiding Jiang, and Zico Kolter. Language models are weak learners. Advances in Neural Information Processing Systems, 2023.
[30] Jaehyun Nam, Woomin Song, Seong Hyeon Park, Jihoon Tack, Sukmin Yun, Jaehyung Kim, and Jinwoo Shin. Semi-supervised tabular classification via in-context learning of large language models. In Workshop on Efficient Systems for Foundation Models@ ICML2023, 2023.
12

---

[31] Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data, Advances in Neural Information Processing Systems, 34: 18932-18943, 2021.
[32] David Bonet, Daniel Mas Montserrat, Xavier Giró-i Nieto, and Alexander G Ioannidis. Hy- perfast: Instant classification for tabular data. AAAI Conference on Artificial Intelligence, 2024.
[33] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 137719111 2020.
[34] Sanh Victor, Webson Albert, Raffel Colin, Bach Stephen, Sutawika Lintang, Alyafeai Zaid, Chaffin Antoine, Stiegler Arnaud, Raja Arun, Dey Manan, et al. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022.
[35] Jachyun Nam, Woomin Song, Seong Hycon Park, Jihoon Tack, Sukmin Yun, Jachyung Kim, Kyu Hwan Oh, and Jinwoo Shin. Tabular transfer learning via prompting llms. Conference Language Modeling, 2024.
[36] Jiahuan Yan, Bo Zheng, Hongxia Xu, Yiheng Zhu, Danny Chen, Jimeng Sun, Jian Wu, and Jintai Chen. Making pre-trained language models great on tabular prediction. In International Conference on Learning Representations, 2024.
[37] Wei Fan, Erheng Zhong. Jing Peng, Olivier Verscheure, Kun Zhang, Jiangtao Ren, Rong Yan, and Qiang Yang. Generalized and heuristic-fre free feature construction for improved accuracy. In Proceedings of the 2010 SIAM International Conference on Data Mining, pages 629-640. SIAM, 2010.
[38] James Max Kanter and Kalyan Veeramachaneni. Deep feature synthesis: Towards automating data science endeavors. In 2015 IEEE international conference on data science and advanced analytics (DSAA), pages 1-10. IEEE, 2015.
[39] Liyao Li, Haobo Wang, Liangyu Zha, Qingyi Huang, Sai Wu, Gang Chen, and Junbo Zhao. Learning a data-driven policy network for pre-training automated feature engineering. In The Eleventh International Conference on Learning Representations, 2022.
[40] Benjamin Doerr and Frank Neumann. A survey on recent progress in the theory of evolution- ary algorithms for discrete optimization. ACM Transactions on Evolutionary Learning and Optimization, 1(4):1-43, 2021.
[41] Nina Mazyavkina, Sergey Sviridov, Sergei Ivanov, and Evgeny Burnaev. Reinforcement learning for combinatorial optimization: A survey, Computers & Operations Research, 134:105400, 2021.
[42] Wei- Vin Loh. Classification and regression trees. Wiley interdisciplinary reviews: data mining and knowledge discovery, 1(1):14-23 22211422,,22 2011.
[43] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. arXiv preprint arX1v:2307.03172, 2023.
[44] Jaehyun Nam, Jihoon Tack, Kyungmin Lee, Hankook Lee, and Jinwoo Shin. Stunt: Few-shot tabular learning with self-gene tasks from unlabeled tables. In International Conference on Learning Representations, 2023.
[45] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
[46] Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint 1423,4 2023.
13

---

[47] Arthur Asuncion and David Newman. Uci machine learning repository, 2007.
[48] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: next-generation hyperparameter optimization framework. Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2623-2633 2019.
[49] Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Pieter Gijsbers, Frank Hutter, Michel Lang, Rafael G Mantovani, Jan N van Rijn, and Joaquin Vanschoren. Openml benchmarking suites, arXiv preprint arXiv:/708.0333/ 2017.
14

---

Appendix: Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning
A Prompt examples
A.1 Generation of a new column
In Listing 1, we present the prompt Pco1, which instructs the LLM generate a new column name Cnew- The prompt includes a detailed explanation of each feature column, specifying its type and values. For convenience, we restricted Cneu to be either binary or categorical.
E Your task ###
Your objective is predict (Objective). You have
the following
+ attributes:
[Column Name): (Numerical value of (Column Min) (Column Max))
(Column Name): (Boolean)
(Column Name):
(Categorical value of (Value 1), {Value 2})
To enhance prediction performance, what additional attributes should be considered? These attributes should be either binary (e.g., 'Yes' 'No') or categorical (e.g- 'high', 'low', 'moderate'). Please propose 4 attribute that is not listed above.
## Answer ###
Listing
Prompt for the generation of a new column Crav-
A.2 Rule initialization
In Listing 2, present the prompt Pinit. which instructs the LLM to create an initial rule for generating new column Czex- The LLM generates a new column, Coews by considering the existing column features in the dataset and the semantic meaning of the name of Czex
f'iigg Your task ###
You have access to the following attributes:
[Column Name): (lumerical value of (Column Min) {Column Max}+)
[Column Name): (Boolean)
[Column Name): (Categorical value of (Value 1), {Value 2}, (Value 3})
### Question ###
Give a good rule to predict the 'New Column Name} (<Output Category or + (Output Category 2}) with the attributes listed above.
#* Answer ###
Listing 2: Prompt for the rule initialization.
15

---

A.3 Generation of a rule
In Listing 3, we present the prompt Pgen. which instructs the LLM generate an improved rule compared to those previously generated. The prompt includes a list of (rule, tree- baseb reasoning, score) tuples. Since the score represents the accuracy of the XGBoost classifier, the trajectory sorted in ascending order of score. For regression tasks using mean absolute error, the trajectory is sorted in descending order.
f'''I have rules predict (Objective) with attributes listed below.
[Column Name} (Numerical value of {Column Min] (Column 1 Max))) {Column
~ Feature Description)
(Column Name): (Boolean): {Column 2 Feature Description)
(Column Name) (Categorical value of (Value 1), (Value 2), (Value 3)); (Column
Feature Description)
We also have corresponding decision trees (CART) to predict (Objective) from the attributes listed above along with predicted (New Column Name]. The rules arranged in ascending order based their scores evaluated with + XGBoost classifier, where higher scores indicate better quality.
Rule to predict (Objective):
[Rule 1}
Decision tree (CART):
(Decision Tree 1}
Score evaluated with XCBoost classifier: {Score 1}
to predict (Objective):
(Rule 2}
Decision tree (CART):
(Decision Tree 2} Score evaluated with XGBoost classifier:
(Score
Give new rule to predict (Objective) different from old ones (but + should use the listed attributes above) and has score as high as possible.
Improved !!!e:!!!
Listing 3: Prompt for the rule generation.
16

---

A.4 Translation of a rule into Python code
In Listing 4, we present the prompt Peode. which instructs the LLM to translate the rule into Python code. To minimize potential syntactic enrors, we impose several restrictions. First, we explicitly define the variable types wherever possible. We also instruct the LLM account for the feature value types before performing calculations, e.g., avoid the addition of categorical and numerical values.
Rule
[Rule to translate)
## Your Task #gg
Change rule into executable Python code. Consider the type of each feature. Input data (Kumpy array (not a Dict)): [fall_column_naat
[Column Name): (Numerical value of [Column Min} (Column Max})
[Column Name): (Boolean)
{Column Name): (Categorical value of (Value 1), (Value 2})
Output: (Output Category 1} (Output Category 2}
Give instantly executable code without example usage. Consider the feature value types (avoid calculating categorical value and numerical value), Start 'def (Rule Function Name) (data):
+
+
### Only python code ###
Listing 4: Prompt for translating rule in natural language into Python code.
17

---

B Dataset details
In this section, we provide further details on the datasets.
B.1 Datasets with language descriptions
1. Disease!
classification task predict patient's disease diagnosis based on the following attributes:
Binary (Yes or No): Fever, Fatigue
Numerical (Range): Age (25 90)
Gender (Female or Male), Blood Pressure (High, Normal, or Low), Cholesterol Level (High, Normal, or Low)
Categorical:
Clinical Trial
A classification task to predict patient mortality in clinical trials based on the following attributes:
Binary (Yes or No) Historical Disease: Deep Vein Thrombosis, Pulmonary Embolism, Antiandrogen Therapy, Cardiac Failure, Respiratory Failure, Venous Insufficiency, Coronary Artery Disease, Myocardial Infarction, Hypertension, Peripheral Arterial Occlusive Disease
Binary (Yes or No) Medication:
Dexamethasone, Ondansetron, Heparin, Fluorouracil, Ranitidine, Cisplatin, Metoclopramide, Carboplatin, Furosemide
Academic
A classification task to predict whether a student would dropout based on the following attributes:
Numerical: Marital Status, Daytime/Evening Attendance, Previous Qualification, Nation- ality, Father's Qualification, Father's Occupation, Displaced, Debtor, Tuition Fees up to Date, Gender, Scholarship Holder, Age at Enrollment, International, Curricular Units 1st Sem (Approved), Curricular Units 1tt Sem (Grade), Curricular Units 2nd Sem (Approved), Curricular Units 2nd Sem (Grade).
4. Enefit
regression task predict daily energy consumption based the following attributes:
Prediction Unit Day, Hour, Lowest Price Per MWh, Highest Price Per MWh, Installed Capacity, Euros Per MWh, Local Forecast Temperature, Local Forecast Dewpoint, Local Forecast Cloudcover Total, Local Forecast Metre U Wind Component, Local Forecast 10 Metre V Wind Component, Local Forecast Direct Solar Radiation, Local Forecast Surface Solar Radiation Downwards, Local Forecast Total Precipitation.
Numerical:
Tesla Stock
A regression task to predict the target day's highest stock price based on the following attributes:
Numerical:
Open Price of 2 Days Before, Highest Price of 2 Days Before, Lowest Price of Days Before, Close Price of Days Before, Open Price of Day Before, Highest Price of Day Before, Lowest Price of Day Before, Close Price of 1 Day Before, Open Price of the Target Day, Time Index.
"https///www. http:///www.....n.ooommmaaaanaao
patient--rroile-daaasse
3 hhtps://www. wttp://wwww...cooeoooooooooonnno
4 /https://www http://wwwwwww/c.....oo/o wtwp////mmmcioooo////ooeeeeeeeeo htwp:///cc.............oooooooom
ww httpp://wwww..gggg..cm///aasssss www.p//gw....eeaeee/e//lleeeeee.
18

---

B.2 Datasets without language deseriptions
For our main results on datasets without language descriptions (see Table 3), we evaluate the 19 classification datasets from the tabular benchmark proposed by Grinsztajn et al. [13]. Following the curation approach from Grinsztajn et al. [13]), we uniformly subsample to 50,000 instances for datasets exceeding this size. We provide brief dataset statistics below.
Table Dataset statisties. 19 classification datasets benchmarked by Grinsztajn al. [13].
Dataset
OpenMI
Samples
Features
44160
4970
electricity
44156
38474
compass
44162
16644
17
wine
44091
2554
11
house_ 16H
44123
13488
MagicTelesscop (Magic)
44125
13376
Higgs
44129
940160
24
jannis
44131
57580
54
credit
44089
16714
10
eye movements
44157
7608
MudCCu099uupsllling (kddCup09)
44158
5032
45
road-safety
44161
111762
32
bank-marketing
44126
10578
phoneme
44127
3172
covertype
44159
423680
california
44090
20634
kdd_ udd_ipums__i Laams_la__99____ 1a_97-small feddd_ppmm_____
44124
5188
20
MiniBooNE
44128
72998
50
pol
44122
10082
C Baseline details
In this section, we describe the hyperparameter search space for the baseline models. For each random split every dataset, we find the optimal set of hyperparameters using random sampler run for 400 trials. We utilize the Optuna library [48] for the hyperparamter tuning.
C.1 XGBoost
For XGBoost, we adopt the hyperparameter search space used in Grinsztajn et al. [13].
Table 10: XGBoost [11]
hyperparameters space.
Parameter
Distribution
Max depth
UniformInt [1, 11]
Num estimators
UniformInt [100, 6100, 200]
Min child weight
LogUniformmnt [1, le2]
Subsample
Uniform [0.5, 1]
Learning rate
LogUniform [1e-5, 0.7]
Col sample by level
Uniform [0.5. 1]
Col sample by tree
Uniform [0.5, 1]
Gamma
LogUniform [le-8, 7]
Lambda
LogUnifome [1, 4]
Alpha
LogUniform [le-8, 1e2]
19

---

C.2 Multilayer perception (MLP)
For MLP, we adopt the hyperparameter search space from Grinsztajn et [13] and the architecture from Gorishniy et al. [31], which includes learning embeddings for categorical features. The models trained for up 300 epochs with early stopping, with the model that achieves the best validation score selected for evaluation. If validation scores do not improve for 40 epochs, training is stopped early. For learning rate scheduling, we use PyTorch's 's ReduceOnPlateen implementation.
Table 11: MLP [31] hyperparameters space.
Parameter
Distribution
Num layers
UniformInt [1, 8]
Layer size
Uniformint [16, 1024]
Dropout
Uniform [0, 0.5]
Learning rate
LogUniform [le-5, 1e-2]
Category embedding size
Uniformint [64, 512]
Learning rate scheduler
[True, False] [256, 512, 1024]
Batch size
C.3 HyperFast
For HyperFast [32], we adopt the hyperparameter space from the original paper.
Table 12: HyperFast [32] hyperparameters space.
Parameter
Distribution
N ensemble
[1, 4, 8, 16, 32]
Batch size
[1024, 2048]
NN bias
[True, False]
Stratify sampling
[True, False]
Optimization
[None, 'optimize", gesembleeopttmizze
Optimize steps
[1. 4, 8, 16, 32, 64, 128]
Seed
UniformInt [0, 9]
D
Compute resources
We conducted our experiments on variety of machines, including
CPU: Intel(R) Xeon(R) Gold 6226R, GPU: RTX 3090
CPU: Intel(R) Xeon(R) Gold 6426Y, GPU: RTX 4090
CPU: Intel(R) Xeon(R) Gold 6426Y, GPU: RTX A6000
20

---

E Comparison with CAAFE
E.1
Main results
Table 13: Performance improvements by OCTree on datasets with language descriptions. We report test етrоr rates (%) on six classification tasks (*) and mean absolute errors (x10-* for two regression tasks (f). The lowest enror is in bold. Values in parentheses indicate the relative error rate reduction from the baseline. We report the mean error and standard deviation across three random splits, except for the two regression tasks (time series tabular data), which are split by time index.
GPT-4o was used for both CAAFE and OCTree.
Dataset
Baseline
CAAFE [19]
OCTree (Ours)
Tesla
6.61
6,36 (3.8%)
5.48 (17.1%)
Enefit
8.00
7.97 (0.4%)
7.82 2.3%)
Disease"
28.09479
27.46276 (2.2%)
25.72:66
8.4%)
Clinical"
46.27:.5.
45.39 (1.9%)
43.75
5,4%)
Academic
14.15106
14.13 (0.1%)
13.74
2.9%)
BTC
25.11
24.67 (1.8%)
24.00
4,4%)
Diabetes*
5.45436
4.92+18 (9.8%)
4.16239
(23.6%)
Student"
36.17120
36.00=20 (0.5%)
35.83 0.9%)
We conduct a comparative evaluation using all datasets (along with additional ones) with contextual information from the experiments summarized in Table 1. First, CAAFE [19] exhibited high variance, even the same dataset split, partly due the randomness associated with GPT-4o's 's temperature- based sampling. At times, it failed to improve upon the baseline. To address this, we average the performance over three trials per random split and report the mean and variance. As shown in Table 13, our method consistently outperforms CAAFE. Note the official implementation of CAAFE has been slightly modified accommodate regression tasks.
E.2 Case study
f!r1
df ('Ae Category'] pd. cut (df ['Age'], bins- [0, 30, 60, 100], labels- [ ('Young', 'Adult', 'Senior'])
+
[f'Fever_Cough Interactio ']
df ['Fever'] df Cough' ']
Listing 5: New columns introduced by CAAFE [19].
frr
If the individual has "Fever" "Fatigue" and "Difficulty Breathing" with "Age" 4 between 60 and 90, then predict "Exposure to Infected Individuals" as "Yes", Otherwise, predict "Exposure to Infected Individuals" as "No".
Listing 6: New columns introduced by OCTree (Ours).
As illustrated in Listings and CAAFE struggles to introduce meaningful columns for the disease dataset, whereas our method proposes more relevant and coherent rules.
21

---

Case studies on different types of LLMs
In Section 4.2, we evaluate three different LLMs and provide additional comparisons this section. As shown in Table 4, our custom model (the Llama 2 model trained fine- tene on high-quality open dialogue dataset) achieves the lowest error rate, followed by Code Llama and the Llama chat base model, both of which still outperform the baseline XGBoost. We also observed several notable differences in feature generation patterns among models.
First of all, Code Llama utilizes various NumPy operations (e.g., 'np.sin') (riin)) to generate mathematically sophisticated features:
x9 ((x4 + 0.24) x1) + ((x5 + 0.27) * x2)
x9 = np.sin(x1) np.cos(x4)
x9 x4 np. tx)8))
Secondly, Llama 2 chat base favors simple polynomial combinations:
x9 = x4 x1 * (x2 + x6) +2
(x2 x6) +4 (x7 + x8)
x6) +3 (x7 + x8) (1 x5 +4)
Finally, our custom model tends to explore polynomial space of features while also utilizing built-in Python functions such as 'abs()'. Compared to the Llama chat base model, it more effectively navigates a broader space of features:
x9 x1
+2 (x2 x3)
abs x1 *x1 x2 x4 x5 x7 x8
x9
x1 (x1 + 2) x2 * (x2 0.5) (x2 0.5)
These observations suggest that, while all three models can generate useful features, our custom model more effectively navigates a broader feature space, leading the generation of even valuable features. We suspect that further training on code data, particularly to enhance the ability to leverage scientific computing libraries like NumPy, could lead to additional performance improvements.
G Scalability of OCTree
Here, we evaluate the scalability of our method on datasets with hundreds of features (e.g.. 501) from the OpenML repository [49]. We choose XGBoost [1]] as the baseline model, because it is the most competitive baseline in our main experiments (see Table 3). Additionally, we - as the rule generator, the Llama 2-based model we primarily use is constrained by relatively short context length, which becomes limiting as the prompt size in- creases with the number of features. As shown in Table 14, our method scales effectively to datasets with larger number of features. For example, on the madelon dataset, which contains 501 columns, method reduces the relative error by 6.3% compared to the XGBoost baseline.
Table 14: OCTree on datasets with hundreds of features. We report the the lowest error is highlighted in bold. Values in parentheses indicate the relative error reduction from the baseline model (i.e., XGBoost [1]]).
error (%) and
Dataset
# features
Baseline
OCTree (Ours)
madelon nomao
501
21.54
20.19 (6.3%)
119
3.08
2.84 (7.8%)
22

---

H Examples of rule optimization
To identify an effective feature generation rule, we conduct 50 rounds of optimization with the LLM. In each round, the LLM is provided with an optimization trajectory that includes reasoning information from previous experiments. Consequently, the imput prompt evolves throughout the optimization process, with later rounds incorporating more accumulated information. Below, we show the first and last five output rules generated during the optimization on the electricity dataset.
First five:
x3 (x5 x8) x6
x12 x5 x2 (x3 x4 x6) x1
x12 = x11 + (x7 x10) * x2 x8
x12 = x3 + (x6 + x8 xx x2) x1
x12 = (x1 x6 x2 + x8) * x3 x7
Last five:
x12 = ((x2 x1) (x11 x4)) x2 (0.5 (x1 0.3))
x12 = ((x2 x1) (x11 x4)) +6
x12 ((x2 x1) (x11 x4)) #14 (0.02 (x5 x7)) 0.5
x12 (x11 x1) (x11 (x6 x7))
x12 ((x2 x1) (x11 x4)) +3 (0.1 (x5 x6)) +2
Note that in the early stages optimization, the LLM tends to explore wider variety of rules, allowing for greater exploration. In contrast, during the later stages, the model narrows its focus to refine features within a more constrained space.
I Broader impacts
Our method is particularly effective in scenarios where collecting real data is costly restricted, such as in the finance or medical domains, where data availability is often limited due to privacy concerns. However, since the features generated by OCTree are artificial, crucial carefully inspect these features for their relevance and reliability.
23

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims in the introduction and abstract accurately reflect the contribution and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
We do not have a theory in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide implementation details including hyperparameter search space of prediction models in Appendix C. We also provide where the datasets are from in Appendix B. Additionally, we provide all prompts we use in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
25

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide the URL the code the Abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips) cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ttpp..ccppbblll/ http:ccccpppccccccecceeccccccccc ctppublicgguuieeesseeeeppp guides/ /CoeeSubmdeabiii issi ssisssoolooooooooo snPoicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the detail data splits in Section 4, hyperparameters (including how they chosen, and optimizers) in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All experiments are conducted with the same and commonly used random seed, and report standard deviation across random splits.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
26

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide compute resources we used in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : /nwrrisrrpp..e/ wc/ppublic/thiiceuuuut
Answer: [Yes]
Justification:
We do not have any ethical concerns.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the societal impact Appendix I.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Our framework does not introduce risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited all papers and datasets in Reference and Appendix B.
Guidelines:
existing assets. The authors should cite the original paper that produced the code package or dataset.
The answer NA means that the paper does
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
28

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not have human subject.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
29


Reviews:
[{"review_id": "W2cmu3NMVv", "reviewer": "Reviewer_6JM8", "summary": "The authors introduce a novel feature engineering technique that leverages LLMs for language-based reasoning and considers the outcomes of past experiments as feedback for iterative rule improvements.", "questions": "We consider the paper interesting and relevant. Nevertheless, we would like to point to the following improvement opportunities:\n\nGENERAL COMMENTS\n\n(1) - The authors compare their model against models on which such feature generation was not applied, showing promising results. Nevertheless, we miss a comparison against similar methods that generate features either with heuristics or LLMs, as explained in the Related Work section. In particular, we would encourage the authors to include some comparison against the following work: Hollmann, Noah, Samuel Müller, and Frank Hutter. \"Large language models for automated data science: Introducing caafe for context-aware automated feature engineering.\" Advances in Neural Information Processing Systems 36 (2024).\n\n(2) - Did the authors execute several rounds of feature generation using the same prompt and input? Did they observe any variability regarding the results obtained?\n\n(3) - How are results different when considering different LLMs? Are there certain patterns regarding the features generated/not generated? Furthermore, the authors mention that \"incorporating code datasets alongside dialogue datasets may further enhance the performance of the rule generator LLM\". How were such prompt results better from regular LLMs and different from those generated by the LLM proposed by the authors (e.g., in Table 3)?\n\n(4) - Did the authors face LLM hallucinations? We would appreciate a brief description on how did they handle such cases and some quantification, to understand how frequent such problem is across different LLMs they tried. \n\n\nFIGURES\n\n(6) - Figure 3: (i) can be made monochromatic, (ii) are datasets balanced? How is the cut threshold determined to measure Accuracy? Could this be replaced with a threshold-independent metric? \n\nTABLES\n\n(7) - Table 7: bold the best results", "limitations": "The authors acknowledge some limitations of their work. Nevertheless, we consider some additional weaknesses (see weaknesses section) should be addressed and eventually acknowledged.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- the authors introduce a novel feature engineering technique that leverages LLMs so that past experience can be considered when proposing novel features.\n - the paper is well-structured, the results are clearly presented, and multiple LLMs are assessed", "weaknesses": "- the authors did not compare their method against other techniques for automated feature generation\n- the authors provide no details on whether several rounds of LLM prompting were executed, how do results vary across multiple executions and whether they had issues with LLM hallucinations", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 6JM8,\n\nWe sincerely appreciate the time and effort you dedicated to reviewing our manuscript and providing insightful comments. Below, we address each of your points individually.\n\n---\n**[W1, Q1] The authors did not compare their method to other feature generation techniques, in particular, CAAFE [1].**\n\nWe want to first clarify that we have already compared our method with other techniques for automated feature generation, such as AutoFeat [2] and OpenFE [3], and verified that our method performs considerably better (see Table 4 in our manuscript). Furthermore, unlike competing methods, our approach generates semantically meaningful features that human annotators can use to provide additional labels, which can significantly enhance model performance. For example, as shown in Figure 3, collecting real values for the suggested features significantly improves patient mortality prediction.\n\nTo address your concern further, we compared our method to CAAFE. Please refer to the global response above for additional details. We thank the reviewer for the suggestion and will incorporate the results in the final manuscript.\n\n---\n**[W2-1, Q2] Number of prompting rounds and results across multiple executions.**\n\nTo find an effective feature generation rule, we execute 50 rounds of LLM prompting. In each round, LLM is provided with an optimization trajectory, including reasoning information highlighting past experiments. Consequently, the input prompts vary across rounds, with later rounds incorporating more accumulated information.\n\nTo further address your comment, we describe how the output rules evolve throughout the optimization rounds on the electricity dataset. Due to space constraints, we show the first and last five output rules below. In the early optimization stages, the LLM produces diverse outputs, suggesting active exploration of possible rules. In contrast, during the later stages, the LLM refines the solution space around previously discovered solutions, making only minor adjustments. We hope this additional analysis provides deeper insight into our method.\n\nFirst five:\n- x12 = x3 + (x5 * x8) * x2 - x6\n- x12 = x5 * x2 + (x3 * x4 - x6) - x1\n- x12 = x11 + (x7 * x10) * x2 - x8\n- x12 = x3 + (x6 * x8 * x2) - x1\n- x12 = (x1 * x6 + x2 * x8) * x3 - x7\n\nLast five:\n- x12 = ((x2 - x1) * (x11 - x4)) ** 2 - (0.5 * (x1 - 0.3))\n- x12 = ((x2 - x1) * (x11 - x4)) ** 6\n- x12 = ((x2 - x1) * (x11 - x4)) ** 14 - (0.02 * (x5 - x7)) - 0.5\n- x12 = (x11 - x1) + (x11 * (x6 - x7))\n- x12 = ((x2 - x1) * (x11 - x4)) ** 3 + (0.1 * (x5 - x6)) ** 2\n\n---\n**[W2-2, Q4] Handling hallucinations and their frequency across different LLMs.**\n\nWhile it is true that the LLM can occasionally suggest suboptimal or semantically incoherent rules, our method is designed to overcome these hallucinations. Specifically, we provide feedback on previously generated rules to guide the LLM to iteratively improve its rule generation. This feedback loop helps the LLM avoid hallucinations, which have likely resulted in low validation scores in subsequent iterations. Empirically, we have found that these issues occur more frequently in the early stages when the LLM explores the rule space more extensively and for less capable LLMs, e.g., those without additional training on dialogue or code generation data.\n\n---\n**[Q3] Case studies for different LLMs (e.g., results, patterns).**\n\nIn our ablation study, we evaluated three different LLMs. As shown in Table 3 of our manuscript, our custom model (the Llama 2 trained additionally on a high-quality dialogue dataset) achieved the lowest average error rate, followed by the Code Llama and then the Llama 2 chat base, which still outperformed the baseline XGBoost. In terms of feature generation patterns, we observed several differences between the models.\n\nCode Llama: This model utilizes various numpy operations (e.g., `np.sin`), which can generate mathematically sophisticated features.\n- x9 = ((x4+0.24) * x1) + ((x5+0.27) * x2)\n- x9 = np.sin(x1) * np.cos(x4)\n- x9 = x4 * np.tan(x8)\n\nLlama 2 chat base: This model favors various polynomial combinations.\n- x9 = x4 * x1 * (x2 + x6) ** 2\n- x9 = x4 * x1 ** 3 * (x2 + x6) ** 4 * (x7 + x8)\n- x9 = x4 * x1 ** 2 * (x2 + x6) ** 3 * (x7 + x8) * (1 + x5 ** 4)\n\nOurs: Our custom model also tends to explore polynomial space of features, often using built-in Python functions such as `abs()`. However, our model explores a considerably broader space of features more effectively than the Llama 2 chat base.\n- x9 = x1 ** 2 * (x2 - x3)\n- x9 = abs(x1) ** x1 + x2 - x3 - x4 - x5 - x6 - x7 - x8\n- x9 = x1 * (x1 + 2) - x2 * (x2 - 0.5) * (x2 - 0.5)\n\nThese observations suggest that while all three models have strengths in feature generation, our custom model's broader exploration and more effective use of the feature space contribute to its superior performance. However, we believe equipping our custom model with enhanced code generation capabilities, such as utilizing scientific computing libraries like numpy, could lead to even more performance gains. We will incorporate these points into our final manuscript.\n\n---\n**[Q6-2] Figure 3: Details on the dataset and evaluation with various metrics.**\n\nThe dataset used in Figure 3 is nearly balanced with a class distribution of 55:45. Since it is a binary classification, the model outputs the class with the higher probability as the prediction result (i.e., no cut threshold). While accuracy is a suitable metric, we also report the ROC-AUC, a threshold-independent metric, in Figure 1 in the attached PDF to provide additional insights. Here, we highlight that using the real values improves all metrics, demonstrating that OCTree effectively recommends useful columns for the target task.\n\n---\n**[Q6-1, 7] Editorial comments on Figure 3 and Table 7.**\n\nThanks for your feedback. In the final draft, we will revise Figure 3 to be monochromatic (as in Figure 1 in the attached PDF) and make the best results bold in Table 7.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MeFIQhapgV", "reviewer": "Reviewer_Pmbp", "summary": "This paper proposes a new tabular learning framework called OCTree (Optimizing Column Feature Generator with Decision Tree Reasoning). The framework leverages the reasoning capabilities of large language models (LLMs) to automatically generate new column features based on feedback from decision trees. Experimental results demonstrate that this framework consistently outperforms existing automatic feature engineering methods across various tabular data prediction tasks.\nThe OCTree framework effectively utilizes the reasoning capabilities of LLMs and feedback from decision trees to automatically generate new column features, significantly enhancing the performance of tabular data prediction tasks. This framework demonstrates the great potential of LLMs in automatic feature engineering for tabular data.", "questions": "see weakness", "limitations": "see weakness", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "S1: A new framework for automatic feature generation that leverages LLMs' language understanding and reasoning capabilities, using feedback from decision trees.\n\nS2: All experiments on various real-world datasets show significant performance improvements in different prediction models, including gradient-boosted decision trees and deep neural networks.\n\nS3: The method works for both datasets with and without language descriptions and shows good transferability of generated features across different types of models.", "weaknesses": "w1: I thoroughly enjoyed reading this work. The authors present a unique perspective on the optimization of tabular data tasks. For example, in the abstract, the authors highlight the shortcomings of existing work, such as \"they often rely solely on validation scores to select good features, neglecting valuable feedback from past experiments that could inform the planning of future experiments.\" This indeed reflects the current issues faced by Automated Feature Transformation, where high-performing features may not necessarily be meaningful features. However, conversely, it seems that the goal of Feature Engineering is to improve the performance of downstream machine learning tasks, which is a very direct objective. Isn't it contrary to the original intent of feature engineering if validation scores are not used as an evaluation metric?\n\nw2: The writing of the entire paper is relatively easy to read, although some concepts were not discussed in depth. For instance, the authors only claim that validation scores are a flawed feedback mechanism but do not explore some of the newer methods in the main text, such as the latest approaches utilizing reinforcement learning [1-2] or generative artificial intelligence [3-4]. These methods employ the performance of models like decision trees and other machine learning models as scores to learn strategies.\n\nw3: In Figure 1, Fever, Fatigue, Breathing, etc., are used through processes like Tabular Data Understanding and rule generation to combine boolean forms (Fever, Fatigue) or one-hot forms of Breathing to create a new feature, Smoke. This is quite intuitive; however, if the rules generated in step 1 are inaccurate, the generated column might also be inaccurate. Moreover, can the concept of whether one smokes or not be automatically and adaptively formed during the generation of decision trees or the training of neural networks, rather than requiring such an inaccurate combination? If not, do the authors have experiments or specific mathematical proofs to validate this claim or to show that neural networks, for instance, would converge more difficultly without these generated key features?\n\nw4: In the Column name generation section of the Methodology (section 3.2), how do the authors ensure that the generated column names are valid?\n\nw5: In Table 2, it seems that some datasets with non-linguistic descriptions did not improve or only showed general improvement. What are the specific reasons for this?\n\nAlthough this work seems overly intuitive at times, I remain excited about the application of large language models in this field. I hope the authors can address the above questions.\n\n[1] Traceable group-wise self-optimizing feature transformation learning: A dual optimization perspective, TKDD\n[2] Group-wise Reinforcement Feature Generation for Optimal and Explainable Representation Space Reconstruction, KDD\n[3] Reinforcement-Enhanced Autoregressive Feature Transformation: Gradient-steered Search in Continuous Space for Postfix Expressions, NeurIPS\n[4] DIFER: Differentiable Automated Feature Engineering, ICAML", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Pmbp,\n\nWe sincerely appreciate the time and effort you dedicated to reviewing our manuscript and providing insightful comments. Below, we address each of your points individually.\n\n---\n**[W1] Isn’t it contrary to the original intent of feature engineering if validation scores are not used as an evaluation metric?**\n\nWe would like to first clarify that in our framework, validation scores are indeed a key objective for optimization and are provided as feedback to the rule generator LLM. However, instead of providing just the most recent validation score, we additionally provide a history of the scores to help the LLM better understand the optimization landscape and iteratively refine and improve the rules it generates. We also found that incorporating decision tree reasoning as input is beneficial (see Table 5 in our manuscript). This reasoning serves as a summary of the training dataset, giving the LLM a deeper understanding of the data's structure and relationships. In summary, utilizing information such as a history of validation scores and insights into the training data is essential for enabling the rule generator LLM to effectively handle this complex optimization task.\n\n---\n**[W2] In-depth discussion: The authors only claim that validation scores are flawed feedback mechanisms but need to explore some of the newer methods in the main text.**\n\nThanks for your feedback and for pointing out the related work. As we highlighted above, we want to clarify that validation scores are indeed a key objective we optimize for in our framework. However, we have empirically shown that relying solely on validation scores in a greedy manner can be suboptimal. Our findings show that it is crucial to consider sufficient history of the scores to effectively navigate the optimization landscape. Additionally, incorporating inputs like decision tree reasoning can improve feature quality by providing a richer context for the task.\n\nRegarding related methods, framing feature generation as a sequential decision-making problem allows techniques such as reinforcement learning to be applied. Similar to our basic approach, these methods generate feature candidates and evaluate them on downstream tasks for iterative improvement. However, some of these methods require solving multiple Markov decision processes [4, 5], which can be complex and computationally demanding. In contrast, our framework offers a conceptually much simpler approach that is easier to implement. Our experiments show that an open LLM of moderate size can effectively serve as a rule generator across various datasets and prediction models. With optional fine-tuning to enhance chat or code generation capabilities, the model performs well without needing additional training or customization for each specific setting.\n\nWe will incorporate these points more clearly into our manuscript.\n\n---\n**[W3-1] If the rules generated in step 1 are inaccurate, the generated column might also be inaccurate.**\n\nWhile it is true that the rule generator LLM can occasionally suggest suboptimal rules, we provide feedback on previously generated rules, guiding the LLM to iteratively refine and enhance its rule generation. This feedback loop is an important component of our framework, allowing the LLM to converge toward more effective and accurate rules.\n\n---\n**[W3-2] Can the concept automatically be formed during the generation of decision trees or the training of neural networks rather than requiring such an inaccurate combination?**\n\nYou are indeed correct that decision trees inherently represent a disjunction of conjunctions of feature values and that neural networks learn latent features through hidden layers. However, our empirical findings suggest that generating explicit features using our approach and incorporating them as additional inputs yields better results than relying solely on these models to infer implicit features. We suspect this is because providing relevant features as explicit inputs enables these models to allocate their capacity more effectively, focusing their degrees of freedom on learning more sophisticated combinations of features that capture subtle patterns within the data.\n\n---\n**[W3-3] If not, do the authors have experiments to validate this claim?**\n\nWe have indeed conducted empirical evaluations to examine the impact of our generated features on model performance. Specifically, we compared the performance of XGBoost and MLP models trained without additional features (relying solely on the implicit features learned by the models) against models trained with the features generated using our framework as explicit input.\n\nAs shown in Tables 1 and 2 in our manuscript, we have observed performance improvements on most datasets, with some showing relative performance gains of more than 20%. These results show that providing the generated features as explicit input can significantly improve the model's ability to converge and perform well, compared to relying solely on implicit feature learning.\n\n---\n**[W4] Ensuring generated column names are valid.**\n\nWe would like to clarify that we have already verified that the LLM indeed generates valid column names. Specifically, we have confirmed that (i) it is beneficial to use the actual values of the generated columns if they are available, and (ii) when given a candidate for a new column name, the LLM can effectively distinguish between columns that are more relevant to the target task (see Section 4.2 of our manuscript).\n\n---\n**[W5] Table 2: Some datasets did not improve or only showed general improvement. Why?**\n\nWe hypothesize that the limited effectiveness observed with some datasets with non-linguistic descriptions stems from the absence of meaningful semantic information that the LLM could leverage during rule generation. However, we would like to emphasize that for most datasets, the generated features enhanced the performance of the prediction models evaluated.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z7YWDJ3mGe", "reviewer": "Reviewer_245r", "summary": "The authors propose an automatic feature engineering method called OCTree. The OCTree algorithm uses LLMs to generate new features. The feature is then used in the training of the black box, and the new validation score is stored. The algorithm uses a decision tree to learn rules that show the reasoning behind selecting these new features and incorporate this in an iterative process. \n\n\nNOTE: If the authors address the issues I have raised in limitations and answer my questions, I will improve my score.", "questions": "Why have the authors excluded CAAFE? I looked at the tutorials, and they seem very easy to use. \n\nWhy do the authors say that CAAFE is orthogonal to their approach? I do not think that is the case. I understand that your approach is CAAFE + a decision tree process embedded. Can you elaborate on the difference between your approach and CAAFE with a concrete example?  \n\nYour notations are so confusing. In Line 131, what does D + r mean? D is a dataset, and r is a rule; they cannot be summed up. \n\nCan authors present evidence on how the inclusion of rules learned by decision trees is increasing the efficiency of their algorithm?", "limitations": "The main competitive algorithm in this work, CAAFE, is excluded from the study. Therefore, we do not have a good baseline for how this approach can improve the problem.   \n\nThe authors have not addressed the study's main concern: the risk of memorization. LLMs are possibly trained on the open datasets they are using. This could be that the LLM is, for example, summarizing what most Kaggle blogs and Medium posts on these datasets have done.", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "The paper's research question, automatic feature engineering, is an important area of machine learning research that has not gained enough attention.", "weaknesses": "It is very difficult for me to follow the notation and understand what the algorithm is doing. This gets harder as the authors have not released the code for the experiments. \n\nThe authors keep iterating that the main limitation of another feature engineering is that they require setting the manual search space without showing how their approach is solving this limitation.\n\nThe experiments do not include CAAFE, the most relevant algorithm to OCTree. The authors claim that the reason behind this is that CAAFE relies on a language-based context of the dataset, making it inapplicable to the datasets they have used. But I am not yet convinced. \n\nNone of the decision-tree reasoning and how it is improving OCTree is explained in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 245r,\n\nWe sincerely appreciate the time and effort you dedicated to reviewing our manuscript and providing insightful comments. Below, we address each of your points individually.\n\n---\n**[W1] Difficulty following the notation and the algorithm. Authors should release the code.**\n\nThank you for your feedback. While several reviewers have noted that the paper is generally well-structured and clearly written, we acknowledge that some notations could be clearer. We will review and clarify these notations to improve understanding.\n\nRegarding the availability of our code, we plan to release it in full. However, due to anonymity requirements, we are providing a version with any author-identifiable components removed. Per the guidelines, we have submitted the link to the AC, so that it can be made available for your review.\n\n---\n**[W2] How does the proposed method tackle the limitations of other feature engineering methods that require setting the manual search space?**\n\nWe would like to clarify that our approach enables the LLM to automatically generate rules without requiring a predefined optimization space. In contrast, other feature engineering methods, such as OpenFE [3], depend on predefined search spaces (e.g., using predefined operators like + or -) to generate candidates. For example, we use prompts like “Give me a new rule that is different from the old ones and has a score as high as possible,” therefore generating features with the rules we get from LLM’s suggestions alone, without any specification of the search space (see Section A.3 of our manuscript). Consequently, our LLM-based framework fully automates the optimization process without the need for a predefined search space.\n\n---\n**[W3, Q1, Q2, L1] Comparison with CAAFE [1] and why orthogonal?**\n\nWe would first like to clarify that, as mentioned in Section 4.1 of our manuscript, we originally excluded CAAFE from our comparison as it requires a language-based context for the dataset. This requirement limits CAAFE's applicability in cases where language descriptions are not explicitly provided, such as when feature names and values are obfuscated to protect confidentiality, as is common in financial and medical datasets. In contrast, the methods that we evaluated in Table 4, as well as our approach, are context-agnostic and can be applied to datasets without such linguistic descriptions. However, our method does benefit from clear language descriptions, as shown in Table 1, when available.\n\nTo further address your concern, we have compared our method to CAAFE and highlighted some key differences and distinctions. Please refer to the global response above for more details. We thank the reviewer for the suggestion and will incorporate the results in the final manuscript.\n\n---\n**[W4, Q4] Decision tree reasoning: How it improves OCTree is not explained in the paper and needs evidence on how including rules increases the algorithm’s efficiency.**\n\nWe would like to first clarify that decision tree reasoning refers to the language description of a decision tree constructed from the entire dataset, including any newly generated features. We have already conducted an ablation study, detailed in Table 5, to assess the effects of providing decision tree reasoning as feedback to the rule generator LLM. Here, our findings indicate that incorporating decision tree reasoning leads to the generation of higher-quality features, resulting in an additional performance improvement of up to 5%. This is because decision tree reasoning provides valuable insights learned from the entire training set, as it highlights the columns that are considered more significant (as nodes in the tree) and the corresponding values (as thresholds of the nodes) used for prediction. We will incorporate these points more clearly into our final manuscript.\n\n---\n**[Q3] Clarification of notation: Meaning of $D \\oplus r$.**\n\nWe would like to clarify that the notation $D \\oplus r := \\\\{x_i \\oplus r(x_i), y_i \\\\}_{i=1}^{N}$ represents the dataset in which each sample $x_i$ is augmented with an additional feature generated by applying the rule $r$. To address your concerns and enhance clarity, we plan to update this notation to $D^r$ in the final draft.\n\n---\n**[L2] Risk of memorization.**\n\nWe would like to emphasize that our method is completely free from the risk of memorization because it generates new column features that are not present in the original dataset. Even if the LLM has memorized information from open datasets, our approach does not rely on the specific data; instead, it generates and introduces entirely new columns to the dataset.\n\nTo further address your concern regarding the potential impact of the LLM being trained on open datasets, we would like to highlight that most of the datasets used in Table 1 (e.g., Tesla, Enefit, Disease, Clinical) were released after the release of Llama 2, our base LLM. The fact that our method enhances the performance of various baseline models on this dataset demonstrates its capability to generate useful features without relying on dataset-specific information. Moreover, Table 2 presents the results of experiments conducted on datasets that lack semantic information about the columns. We used generic indicators (e.g., 'x1', 'x2') for columns and applied ordinal encoding and a min-max scaler to transform all feature values. Even in this setting, our method was able to enhance the performance of a range of baseline models across datasets of varying sizes and characteristics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0etAexD08n", "reviewer": "Reviewer_8DXj", "summary": "The paper introduces OCTree, a framework that uses LLMs to generate and refine feature generation rules for tabular data. By incorporating decision tree reasoning, OCTree iteratively improves feature generation using feedback from past experiments. The framework enhances the performance of some prediction models, including decision trees and neural networks, and demonstrates improvements on real-world datasets. OCTree works with datasets with or without language descriptions and outperforms existing feature engineering methods, showing flexibility in feature generation for prediction tasks.", "questions": "1. Are there other works that use LLMs for feature generation? If so, how does this approach differ from them?\n2. What is the time cost associated with OCTree? How much slower is it compared to directly training XGBoost? Does the time cost increase significantly with the size of the dataset and the feature dimensionality?", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper combines LLMs with decision tree reasoning to generate features for tabular data, offering a novel approach to feature generation. The paper demonstrates improvements through several experiments, and its flexible framework shows potential benefits in enhancing model performance. The paper is well-structured and clearly written.", "weaknesses": "1. The framework is costly and time-consuming due to the need for training new models for validation during each iteration, and it involves some manual operations, preventing full automation.\n2. Lack of comparisons of training times makes it difficult to assess the efficiency relative to other approaches.\n3. The experiments are limited to datasets with a maximum of 54 features, lacking evidence of effectiveness on high-dimensional data (e.g., datasets with hundreds of features).\n4. Datasets are sampled to sizes smaller than 50,000 instances, raising concerns about effectiveness and scalability on larger datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 8DXj,\n\nWe sincerely appreciate the time and effort you dedicated to reviewing our manuscript and providing insightful comments. Below, we address each of your points individually.\n\n---\n**[W1, 2] The framework is costly and time-consuming (with no training time comparison) and involves some manual operations.**\n\nThank you for your comment. As we described in Section 5 of our manuscript, one potential limitation is that it may be time-consuming if the prediction model requires extensive training. In this regard, we acknowledge that our method could be slower compared to competing methods like OpenFE [3]. Thus, we focused on evaluating the effectiveness of our method rather than the run time. However, once the feature generation rules are optimized, new column features can be generated simply by applying these rules to any additional data that becomes available.\n\nAs we also described in our draft, there are several approaches to further scaling our method:\n- Feature transfer: As demonstrated in Table 6, it is possible to first generate features using a simpler model like XGBoost, and then transfer these features to the more complex target model like MLP. This allows for faster rule generation for more complex prediction models.\n- Use of open LLMs: Unlike methods that rely on proprietary LLMs such as ChatGPT, we demonstrated that open models like Llama 2, with minimal additional fine-tuning, can be highly effective in our feature generation framework. This approach avoids the cost associated with paid APIs.\n- Use of data subsets: We have shown our method is effective across datasets of varying sizes. This suggests that during rule generation, a suitable subset of the training data can be used to accelerate optimization, enabling scalability to larger datasets.\n\nLastly, we would like to point out that the only significant manual task involved is designing an appropriate prompt for the rule generator LLM. The version used in our experiments will be fully released alongside the code.\n\n---\n**[W3] Scalability: Number of features.**\n\nThanks for the valuable suggestion. In response, we evaluate the scalability of our method on datasets with hundreds of features (e.g., 501) using datasets from the OpenML repository. We chose XGBoost as the baseline because it was the most competitive baseline in our main experiments (see Table 2). Additionally, we used GPT-3.5-Turbo as the rule generator because the Llama2-based model we primarily used in our work is constrained by a maximum context length of 2048, which becomes limiting as the prompt size increases with the number of features. However, we emphasize that our method is compatible with an arbitrary LLM as the rule generator, as shown in Tables 1 and 3 of our manuscript.\n\nAs shown in the table below, our method scales effectively to datasets with a larger number of features. For example, on the madelon dataset, which contains 501 columns, our method reduces the relative error by 6.3% compared to the XGBoost. Here, we report test error, with values in parentheses indicating the reduction in relative error rates. We will incorporate these results into the final draft.\n\n\\begin{array}{lccc}\n\\hline\n\\text{Dataset}&\\text{\\\\# features}&\\text{Baseline}&\\textbf{OCTree (Ours)}\\newline\\hline\n\\text{madelon}&501&21.54&\\textbf{20.19 (6.3\\\\%)}\\newline\n\\text{nomao} &119&3.08&\\textbf{2.84 (7.8\\\\%)}\\newline\\hline\n\\end{array}\n\n---\n**[W4] Scalability: Number of samples.**\n\nThanks for the valuable suggestion. Following the suggestion, we evaluate the scalability of our method on larger datasets from the OpenML repository using XGBoost. As shown in the table below, our method scales effectively to datasets of a much larger scale. For example, in the nyc-taxi-green-dec-2016 dataset, which contains over 500,000 samples, our method achieves a 13.7% reduction in relative error compared to the baseline. Here, we report test error, with values in parentheses indicating the relative error reductions. We will incorporate these results into the final draft.\n\n\\begin{array}{lccc}\n\\hline\n\\text{Dataset}&\\text{\\\\# samples}&\\text{Baseline}&\\textbf{OCTree (Ours)}\\newline\\hline\n\\text{nyc-taxi-green-dec-2016}&\\text{581,835}&2.91&\\textbf{2.51 (13.7\\\\%)}\\newline\n\\text{Covertype}&\\text{423,680}&2.79&\\textbf{2.15 (22.9\\\\%)}\\newline\\hline\n\\end{array}\n\n---\n**[Q1] Are there other works that use LLMs for feature generation? How does this approach differ from them?** \n\nAs described in Section 2 of our manuscript, CAAFE [1] proposes a context-aware automatic feature engineering method that leverages LLMs but is distinct from OCTree in several key ways. Please refer to the global response above for more details. We thank the reviewer for the question and will incorporate the results in the final manuscript.\n\n---\n**[Q2] Associated time cost.**\n\nThanks for your question regarding various aspects of our method's efficiency.\n\nFirst, the primary time cost comes from (a) using the LLM to suggest rules and (b) training the prediction model on the suggested features to compute validation scores. Directly training an XGBoost would involve a single training run on the original dataset, whereas our method involves multiple iterations until the optimal features are generated. However, once the rules have been generated, applying these rules to any additional data is trivial.\n\nSecond, while dataset size and feature dimensionality mainly affect the training time of the prediction model, they have minimal effect on LLM inference. Additional features may slightly increase the prompt length, but the overall size of the training dataset has little impact on inference time.\n\nLastly, please refer to the approaches to scaling the method we outlined in our response to [W1, 2]. These include using simpler prediction models and leveraging feature transfer, using open LLMs of moderate size like Llama 2 at the 7B scale, and using a subset of the training data during rule generation to improve efficiency.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers and ACs,

We sincerely appreciate the time and effort you dedicated to reviewing our manuscript and providing insightful comments. Below, we discuss some of the common points made by reviewers.

---
**Comparison with CAAFE [1].**

**Restricted applicability of CAAFE.** We would first like to clarify that, as mentioned in Section 4.1 of our manuscript, we originally excluded CAAFE from our comparison as it requires a language-based context for the dataset. This requirement limits CAAFE's applicability in cases where language descriptions are not explicitly provided, such as when feature names and values are obfuscated to protect confidentiality, as is common in financial and medical datasets. In contrast, the methods that we evaluated in Table 4, as well as our approach, are context-agnostic and can be applied to datasets without such linguistic descriptions. However, our method does benefit from clear language descriptions, as shown in Table 1, when available.

**Comparison with CAAFE.** To further address the reviewers’ concerns, we conducted a comparative evaluation using all datasets with contextual information from the experiments summarized in Table 1 of our manuscript. First, we would like to note that CAAFE showed high variance, even on the same dataset split, maybe due to the randomness of GPT-4o's temperature sampling (and sometimes failed to improve the baseline). Therefore, we first averaged the performance of three trials per random split. Then, we report the mean and variance of the three random splits. As shown in Table 1 of the attached PDF, our method consistently outperforms CAAFE (note that we slightly modified the official code of CAAFE to implement regression tasks). Notably, our approach using the open-source Llama 2 model fine-tuned on dialogue data outperforms CAAFE even when it employs the GPT family of models.

**Why is our approach better?** First, we would like to clarify the key distinctions between CAAFE and OCTree. Our approach generates much more semantically meaningful column names (e.g., ‘Smoking Status’), which serve as the basis for creating additional, high-quality columns. By leveraging the LLM’s reasoning, optimization, and in-context learning capabilities, we provide a history of validation scores for candidate features, along with decision tree reasoning, to guide the LLM in effectively navigating the feature space and generating relevant and coherent rules. In contrast, CAAFE primarily relies on the LLM’s language understanding capabilities to suggest simple combinations of existing features, such as binary feature crosses. CAAFE also tends to add new features in a greedy manner, evaluating the validation score for a candidate feature once and discarding it immediately if there is no improvement. Our approach, however, iteratively optimizes each candidate feature, leading to a more effective exploration of potential features.

**Case study.** As illustrated in the example below, CAAFE fails to introduce meaningful columns for the disease dataset, while our method proposes more relevant and coherent rules.

CAAFE:
- df['Age Category'] = pd.cut(df['Age'], bins=[0, 30, 60, 100], labels=['Young', 'Adult', 'Senior'])
- df['Fever_Cough_Interaction'] = df['Fever'] * df['Cough']

Ours:
- If the individual has “Fever” and “Fatigue” and “Difficulty Breathing” with “Age” between 60 and 90, then predict “Exposure to Infected Individuals” as “Yes”. Otherwise, predict “Exposure to Infected Individuals” as “No”. 

**Additional advantages of OCTree.** Moreover, the semantically meaningful features generated by our method enable human annotators to provide additional labels, which can significantly enhance model performance. For instance, as shown in Figure 3 of our manuscript, collecting actual values for the suggested feature substantially improves patient mortality prediction.

**Combining OCTree with other automatic feature engineering methods.** Lastly, we want to highlight that our method is complementary to some of the existing automatic feature engineering techniques. As demonstrated in Table 4 of our manuscript, combining our method with OpenFE [3] leads to further performance improvements. Thus, it is also possible to first generate features with our method and subsequently employ CAAFE to further enhance the feature set.

We thank all the reviewers for the valuable questions and suggestions and will incorporate the results in the final manuscript.

---
**All References.**

[1] Hollman et al., LLMs for Semi-Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering, NeurIPS 2023.

[2] Horn et al., The autofeat Python Library for Automated Feature Engineering Selection, ECMLPKDDW-ADS 2019.

[3] Zhang et al., OpenFE: Automated Feature Generation with Expert-level Performance, ICML 2023.

[4] Wang et al., Group-wise Reinforcement Feature Generation for Optimal and Explainable Representation Space Reconstruction, KDD 2022.

[5] Wang et al., Reinforcement-Enhanced Autoregressive Feature Transformation: Gradient-steered Search in Continuous Space for Postfix Expressions, NeurIPS 2023.

Meta-review: This authors propose a new tabular learning framework called OCTree (Optimizing Column Feature Generator with Decision Tree Reasoning) that leverages the reasoning capabilities of large language models (LLMs) to automatically generate new column features based on feedback from decision trees (i.e., it is an automatic feature space construction method). Methodologically, OCTree algorithm uses LLMs to recommend new features to be generated by name and feature generation rule, which is then used in the training of the ML algorithm for validation. Experimental results demonstrate that OCTree consistently outperforms existing automatic feature engineering methods across various tabular data prediction tasks. 

Consensus strengths of the paper identified by reviewers (and myself) include:
- The idea of using LLMs to generate novel features for ML models over tabular data is conceptually appealing and the specific approach proposed does this in a straightforward way (meant as a positive).
- The paper is very well-written, well-structured, and the empirical results are promising.
- The experiments have a sufficient number of LLMs, baseline algorithms, strong ML models for these tasks, benchmark datasets, and ablation studies to support the core statements claimed in the paper. I actually don't think the reviewers sufficiently appreciate how difficult XGBoost is to beat "out of the box" for these tasks -- which it is so widely used.

Additionally, reviewer concerns regarding comparison to CAAFE and scalability issues were sufficiently addressed during rebuttal.

Conversely, consensus (and my own) observed weaknesses of this work include:
- While issues regarding CAAFE and scalability were addressed during rebuttal, they still need to be integrated into the paper -- but this seems possibly just by editing a revised manuscript.
- Some experimental details (e.g., number of rounds, inter-round feedback) aren't easy to find in the paper (but are in the rebuttal). However, this is easily added.
- While the number of datasets is sufficient, the paper would be non-negligibly strengthened by more datasets, etc. as this would potentially show utility for different cases and could provides cases where interpretability, etc. are more salient (and in line with motivate). Honestly, this would make me more confident in recommending a clear accept.
- While I don't think that the existing experiments were 'polluted' by memorized features (i.e., reviewer 245r), this isn't too difficult to test for (at least for the Kaggle workshop cases) but could be a problem in competition settings -- but likely not an issue in actual deployed systems. Thus, I would include some discussion of this.
- In the same vein, i would add that the generated features are more interpretable than other feature space augmentation algorithms. Of course, generated features are often complex and thus formal interpretation is difficult, but the LLM labels and rules may be useful for this. Discussion is worthwhile as tabular data domains often have this aspect.
- I do agree with reivewer Pmbp that discussion regarding SDP-based feature generation methods is worthwhile; I don't believe empirical comparisons are needed, but this also occurred to me while reading.
- The discussion regarding hallucination in rebuttal should be addressed in the paper. It is clear to me that the ML algorithms can likely perform feature selection (as they have been doing for a long time), but should probably be mentioned.

Overall, I believe this is a nice paper that may have solid practical implications in many 'traditional ML' domains. Additionally, as LLMs continue to improve (including reasoning), I expect this to inspire future work. While there are some details that should be added to the paper, it is well-written overall and shows good performance and convincingly supports the core claims of the work.
Decision: Accept (poster)