Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning

Authors: jaehyun  nam
kyuyoung  kim
seunghyuk  oh
jihoon  tack
jaehyung  kim
jinwoo  shin
  kaist
  trajectory
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 09:35:31

Abstract: In tabular prediction tasks, tree-based models combined with automated feature engineering methods often outperform deep learning approaches that rely on learned representations. While these feature engineering techniques are effective, they typically depend on a pre-defined search space and primarily use validation scores for feature selection, thereby missing valuable insights from previous experiments.
To address these limitations, we propose a novel tabular learning framework that utilizes large language models (LLMs), termed Optimizing Column feature generator with decision Tree reasoning (OCTree). Our key idea is to leverage the reasoning capabilities of LLMs to identify effective feature generation rules without manually specifying the search space and provide language-based reasoning information highlighting past experiments as feedback for iterative rule improvements. We use decision trees to convey this reasoning information, as they can be easily represented in natural language, effectively providing knowledge from prior experiments (i.e., the impact of the generated features on performance) to the LLMs. Our empirical results demonstrate that OCTree consistently enhances the performance of various prediction models across diverse benchmarks, outperforming competing automated feature engineering methods. Code is available at https://github.com/jaehyun513/OCTree.

Main paper: 
 introduction: 
 Learning useful representations from raw data is key to the success of deep learning algorithms, and their effectiveness has been demonstrated across multiple domains, e.g., vision [1, 2, 3, 4] and language [5, 6] . However, in the tabular domain, deep learning approaches are often perceived as less effective [7, 8, 9, 10] . For instance, tree-based approaches utilizing raw column features of tabular data [11, 12] often outperform deep learning models in tabular prediction tasks such as classification and regression [13, 14, 15] . As a result, practitioners commonly resort to using tree-based methods coupled with manual feature engineering, such as computing the product of two column features [16] .Generating suitable column features, even with domain knowledge, can be challenging and costly. For instance, manual validation to identify useful features is infeasible due to the exponentially many possible combinations to explore [17] . To address this issue, existing feature engineering methods [17, 18, 19] use additional filtering schemes [20, 21, 22] to evaluate and select useful features automatically. While these approaches reduce manual effort and improve feature quality, they still present several challenges. First, practitioners often rely on manually defined search spaces to generate candidate features due to the inherent ambiguity of what constitutes informative features [17, 23] . However, this still requires substantial computation for validating candidate features, particularly as the number of features and the complexity of the search space grow. Furthermore, they neglect more effective experimental designs, relying solely on validation scores to select good features, despite the value of past experiment data for improving selection.Motivated by this, we propose to approach this problem from a novel perspective: optimization to discover effective generation rules, leveraging the language understanding and reasoning capabilities of large language models (LLMs). Recent research has demonstrated that LLMs can optimize various non-differentiable problems using prompts that describe the optimization task in natural language [24, 25, 26] . This suggests the potential for LLMs to automatically generate and iteratively refine feature generators without the need for manually specifying the rule space. For example, the reasoning capabilities of LLMs allow incorporating feedback on their previous outputs into the process for iterative refinement. Moreover, linguistic contexts, such as column names (e.g., 'Gender' and 'Age') and categorical values (e.g., 'Female' and 'Male'), could be naturally integrated into the optimization [27, 28, 29, 30] , which is difficult, if not impossible, with conventional methods.Contributions. In this work, we leverage LLMs to generate novel column features for tabular prediction tasks, proposing Optimizing Column feature generator with decision Tree reasoning (OCTree), a generic framework for automated feature generation using LLMs. Figure 1 illustrates an overview of our framework. Our approach begins by prompting an LLM to propose a name for a novel column feature based on the task description, such as 'Trading Volume' for stock price prediction. This initial suggestion guides the LLM in exploring and refining the corresponding feature values. Then, we further leverage the reasoning capability of LLMs to produce a good rule that generates values for the newly introduced column feature based on the existing ones. Specifically, starting from an initial rule r 0 , we let the LLM iteratively improve the current rule r t by using extracted reasonings d 0 , d 1 , . . . , d t and validation scores s 0 , s 1 , . . . , s t attained by the prediction model as feedback. Here, d t denotes the language description of a decision tree fitted to the entire dataset, including the new feature generated by r t . Specifically, we propose using the decision tree reasoning to provide the LLM with effective knowledge from the past experiments, i.e., the prediction model trained with the generated features, providing learned knowledge about the entire dataset. This procedure is iterated for a fixed number of times, after which we select the rule with the highest validation score.We assess the effectiveness of OCTree on a wide range of real-world datasets (e.g., stock price and patient mortality prediction) from various sources, including recent Kaggle competitions. Our experimental results demonstrate that OCTree consistently enhances the performance of various prediction models, including gradient-boosted decision trees [11] and deep neural networks [31, 32] , for both classification and regression tasks. We also assess OCTree on datasets where language descriptions are unavailable, i.e., all feature values and column names are anonymized during preprocessing. Even on these datasets, OCTree reduces relative prediction errors by an average of 5.0% compared to the best baseline model, i.e., XGBoost on the 19 classification tasks benchmarked by Grinsztajn et al. [13] . Here, we use the Llama 2 7B model fine-tuned on high-quality dialogue data to enhance its ability to understand and generate contextually relevant, coherent rules. We also show that OCTree outperforms recent automatic feature engineering methods, including CAAFE [19] and OpenFE [17] , often using our 7B LLM, even when these methods are combined with significantly more advanced models like GPT-4. Lastly, we demonstrate that features generated for one type of model (e.g., a simpler model like XGBoost) can enhance the performance of other model types (e.g., more complex models like neural networks). This illustrates a potential approach to scaling the method for larger, more complex models. 
related work: 
 Tabular learning with LLMs. Recent developments in LLMs have encouraged investigation into their applications to tabular prediction tasks. Dinh et al. [27] and Hegselmann et al. [28] fine-tune GPT-3 [33] and T0 [34] , respectively, by serializing tabular data into natural language. Nam et al. [35] utilizes unlabeled data expressed in natural language for few-shot semi-supervised tabular classification tasks via prompting LLMs. More recently, Yan et al. [36] introduced a tabular-specific tokenization method to pre-train a single language model capable of performing well across multiple tabular datasets. Instead of using LLMs as prediction models, we explore whether they can effectively generate useful column features for tabular prediction tasks. Specifically, we propose enhancing various prediction models by using LLMs as optimizers to generate novel column features.Various prompting techniques have demonstrated the use of LLMs for solving optimization problems. This is achieved by describing optimization problems in natural language and instructing LLMs to iteratively generate new solutions based on previously found solutions and their evaluation. In particular, Yang et al. [25] uses LLMs to optimize linear regression, the traveling salesman problem, and prompt optimization (i.e., refining instructions to improve LLM outputs). Building on these insights, we leverage LLMs to optimize feature generators for tabular prediction tasks. Unlike prior work, we incorporate decision tree reasoning as feedback, providing the model with learned knowledge about the dataset in natural language for more effective optimization. Automated feature engineering. Automated feature engineering involves generating features from raw data without human effort to improve the performance on prediction tasks [19] . Various methods have been developed for this purpose [37, 38, 39] , including iterative feature subsampling with beam search to select informative features [23] and feature boosting and pruning algorithms for efficient and accurate filtering [17] . More recently, Hollmann et al. [19] introduced a context-aware feature engineering approach that leverages LLMs to generate semantically meaningful features based on the description of a given task. Unlike previous approaches, our method leverages the optimization and reasoning capabilities of LLMs to discover effective feature generation rules without the need for manually defining a search space. Furthermore, while methods such as CAAFE [19] rely on language-based context, our approach is applicable to both context-aware and context-agnostic settings, enabling it to handle a broader range of prediction tasks. 
optimizing feature generator with decision tree reasoning: 
 In this section, we introduce a framework for automated tabular feature engineering by leveraging the language understanding and reasoning capabilities of LLMs. In a nutshell, our approach utilizes LLMs as optimizers to propose and refine the rules for generating column features. Specifically, we iteratively improve the rules by guiding the LLMs using (1) the validation performance of previously proposed rules and (2) decision tree-based reasoning derived from the training data as inputs, enabling more effective optimization. We begin by framing the feature engineering problem as an optimization of the rules for generating column features (Section 3.1) and then introduce the core framework, termed Optimizing Column feature generator with decision Tree reasoning (OCTree), designed to solve this optimization task (Section 3.2).Problem setup. Formally, the goal of tabular prediction tasks is to train a prediction model f : X → Y, where X is the input space and x ∈ X is an M -dimensional column feature with corresponding column names C = {c 1 , . . . , c M }. For example, 'x 1 : Female' and 'x 2 : 36' are values for the columns 'c 1 : Gender' and 'c 2 : Age', respectively. In classification tasks, y ∈ Y = {0, 1} K represents the label space with K classes, while in regression tasks, y ∈ Y ⊂ R. We denote by c target the name of the column corresponding to the label y.We frame tabular feature engineering as the optimization of feature-generating rules, where the rules define a mapping from the original set of features to a new feature. Our objective is to generate a one-dimensional column feature, X ′ , by optimizing the rule r : X → X ′ . Specifically, we aim to create a novel column feature that improves the performance of the prediction model when trained with the new feature, f : X ⊕ X ′ → Y. Our optimization problem can be formalized as follows:EQUATIONwhere g is the rule generator (i.e., LLM M in our case), r := g(D train ) is the rule generated based on the training dataset D train , and D ⊕ r := {x i ⊕ r(x i ), y i } N i=1 denotes the dataset augmented with the new column feature generated. L f is the objective function for the given prediction task, such as mean absolute error for regression tasks, evaluated using the model f . In summary, we optimize the rule r to achieve the best validation score measured on D val ⊕ r with the model f * trained to minimize the loss on D train ⊕ r.However, such bi-level optimization is often non-differentiable or computationally demanding, as it involves computing gradients through the optimization of f * . Moreover, the rule itself may involve non-differentiable operations, such as logical conjunctions between categorical features (e.g., 'Is a Smoker = Has Fever ∧ Has Difficulty Breathing' as in Figure 1 ). One approach to addressing the issue is to use black-box optimization methods, such as evolutionary strategies [40] or reinforcement learning [41] . However, these methods also have limitations, including the need for a manually defined search space, which is complex, as well as the potentially suboptimal use of valuable feedback from previously proposed solutions that could enhance the optimization process.To address this issue, we propose leveraging LLMs as optimizers for the tabular feature engineering problem. Our approach involves iteratively proposing and refining rules by prompting LLMs with a trajectory of feedback, which includes the history of previously proposed rules, the validation scores, and the associated reasoning information. Optimization using LLMs [25] has proven to be an effective tool, particularly for black-box optimization problems, which we show can also be effective in tabular feature engineering. Furthermore, LLMs can leverage the semantics of column and feature values for better optimization and provide the flexibility to operate without a pre-defined search space.We now present the core algorithm. First, we prompt the LLM to propose a name of a new column, such as 'Smoking Status' in Figure 1 , and the corresponding rule based on the task description. We then compute the validation score of the prediction model and extract decision tree reasoning from the training dataset, initializing the optimization trajectory. This trajectory provides feedback to the LLM, with the decision tree reasoning component, which effectively conveys the knowledge of past experiments and captures the quality of previously suggested features. As the optimization process continues, the trajectory is updated, enabling the LLM to refine and enhance the rule iteratively.Column name generation. OCTree begins by generating a name of a new column feature, c new , through the LLM M. This is done by prompting M with the prompt p col (see Appendix A.1), which asks for a new column name that would be useful for predicting the target: c new = M(p col (C, c target )). Leveraging its language understanding capabilities, the LLM is able to generate semantically coherent and relevant column names. For instance, it might suggest using trading volume as a new feature for predicting stock prices.Initialize optimization and extract decision tree reasoning. OCTree then generates an initial rule r 0 for deriving a new column feature from the original set of columns C. This is done by prompting the LLM M with the prompt p init (see Appendix A.2) to propose a rule for predicting c new with c 1 , . . . , c M : r 0 = M(p init (C, c new )). The score s 0 for the initial rule is then evaluated with the prediction model f * : s 0 = L f * (D val ⊕ r 0 ). Additionally, decision tree reasoning d 0 is extracted using CART [42] fitted to the training dataset:d 0 = CART(D train ⊕ r 0 ).CART is a binary decision tree that recursively splits the data based on criteria such as Gini impurity to predict the outcome. We employ CART for two main reasons: (i) tree-based models, often ensembles of simple decision trees like CART, outperform deep learning on many tabular prediction tasks, and (ii) CART is easily interpretable and can be expressed in natural language. For example, as illustrated in Figure 1 , CART can be expressed using a simple if-else syntax. Intuitively, the decision tree reasoning extracted by CART provides valuable insights learned from the entire training dataset. It explicitly highlights the columns that are considered more significant (as nodes in the tree) and the corresponding values (as thresholds of the nodes) used for prediction.Optimization with decision tree reasoning. To optimize the rule, we describe the task in natural language and provide the trajectory T t = {(s i , d i , r i )} t i=0 , which includes the history of previously proposed rules, the corresponding scores, and associated reasoning information. We then generate a new rule r t+1 using the LLM M with the prompt p gen (see Appendix A.3), which asks M to propose a new rule that is not present in T t and that would improve on the scores from the previous iterations: r t+1 = M(p gen (T t , C, c target )). The elements in the optimization trajectory are ordered by the scores, as LLMs tend to generate suggestions that appear later in the list [25, 43] . Afterward, we append the new score s t+1 = L f * (D val ⊕ r t+1 ), the decision tree reasoning d t+1 = CART(D train ⊕ r t+1 ), and the rule r t+1 to the trajectory: T t+1 = T t ∪ {(s t+1 , d t+1 , r t+1 )}. The optimization proceeds for a fixed number of iterations, with the best rule selected based on the highest validation score. Generating multiple features. The optimization process can be repeated to generate multiple useful features. For example, after generating the column 'Smoking Status', an additional column 'Physical Activity Level' can be generated based on the original features and the newly created 'Smoking Status'. Formally, we first generate a new column X ′ = r opt (X ), where r opt is the optimized rule for generating the new feature X ′ . This results in an augmented input space, X new = X ⊕ X ′ . Using the updated dataset D new ⊆ X new × Y, OCTree iteratively generates additional column features. This process continues, introducing new features in sequence until the validation score no longer improves. 
experiments: 
 In this section, we evaluate the effectiveness of OCTree across a range of tabular classification and regression tasks using diverse datasets. Our findings demonstrate that OCTree consistently improves the performance of various types of prediction models (Section 4.1 and Section 4.2). Furthermore, ablation studies confirm the value of the proposed decision tree reasoning in the optimization and demonstrate that features generated using one type of prediction model can effectively transfer to others, suggesting an approach to scaling the framework to more complex models (Section 4.3).Datasets. First, we select real-world datasets with language descriptions of the column features from diverse sources, including the Disease, Academic, Enefit, and Tesla Stock datasets, recently released on Kaggle, and the Clinical Trial dataset from the US National Library of Medicine. These prediction tasks are highly practical and relevant to domains such as healthcare (e.g., diagnostics), academia (e.g., student dropout prediction), and finance (e.g., stock price forecasting). In addition, to evaluate OCTree on datasets without language descriptions of the columns, we include the 19 classification datasets benchmarked by Grinsztajn et al. [13] . When selecting datasets, it is important to consider the heterogeneous nature of tabular data [44] and ensure coverage of both categorical and numerical features, as well as both classification and regression tasks. We conduct experiments on this diverse selection of datasets to evaluate OCTree's general applicability across various types of tabular data. Further details on the datasets are provided in Appendix B.Table 1 : Performance improvement by OCTree on datasets with language descriptions. We report test error rates (%) for three classification tasks ( * ) and mean absolute error (×10 -3 ) for two regression tasks ( † ). The lowest errors are highlighted in bold. Values in parentheses indicate the relative error rate reduction from the baseline. We report the mean error and standard deviation across three random splits, except for two regression tasks (time series tabular data), which are split by time index. N/A indicates that the method is not applicable, as HyperFast is a classification model. Baselines. To validate our method, we evaluate it across three types of prediction models. We first consider XGBoost [11] , a highly competitive tree-based model known for its effectiveness in the tabular domain. Second, we apply our method to multilayer perceptron (MLP; Gorishniy et al. [31] ), the base architecture for deep learning models. Lastly, we show that OCTree improves the performance of HyperFast [32] , a recently introduced model designed for fast classification of tabular data. Implementation details, including hyperparameter search space, are provided in Appendix C.Common setup. For all datasets, 60% of the data is used for training, 20% for validation, and 20% for testing. Following Gorishniy et al. [31] , we use learned embeddings for categorical features when training MLPs, while HyperFast handles categorical features automatically. For all experiments, we use CART with a maximum depth of 4 to extract decision tree reasoning, provided to the rule generating LLM in the prompt. Unless noted otherwise, we use the Llama 2 model [45] at the 7B scale, fine-tuned on UltraChat [46] , a dialogue dataset that has been used to develop strong chat models such as UltraLM [46] . Our findings indicate that open models, even at moderate scales, can be highly effective, particularly when equipped with strong chat capabilities. Further comparisons across different types of LLMs are provided in Section 4.2.Datasets with language descriptions. We first experiment with datasets where language descriptions of the columns and categorical feature values are available. In these cases, the LLM generates a logical rule in natural language, which is then converted into Python code for use in our experiments.For this task, we utilize both GPT-4o and our Llama 2 model, as the Llama 2 model can be limited by its relatively short context length on some datasets. See Appendix A.4 for the prompt used.As shown in Table 1 , OCTree consistently enhances the performance of various baseline models. For instance, when generating the column 'Trading Volume' for the Tesla Stock dataset using Llama 2 for XGBoost, the relative error is reduced by 15.9%. OCTree is compatible with arbitrary LLMs, allowing more advanced models to enhance the quality of generated features further. Specifically, with GPT-4o, one of the latest LLMs from OpenAI, our method achieves a relative error reduction of 17.1% on the same dataset for XGBoost. We provide results on additional datasets in Table 13 .Comparison with CAAFE. CAAFE [19] also introduces a feature engineering approach that utilizes LLMs to construct features based on the linguistic context. However, it is important to note that CAAFE requires explicit language descriptions of the features, which limits its applicability when such information is unavailable. For example, feature names and values are often obfuscated for confidentiality, a common practice in the financial and medical domains. In contrast, our method can be effectively applied to datasets without linguistic descriptions, as demonstrated in Table 3 .Table 2 : Applicability and comparison of automated feature engineering methods. We report the mean error (%) and standard deviation across the six datasets with language descriptions used in Tables 1 and 13 Thus, to compare CAAFE with OCTree, we evaluate both methods on datasets with contextual information, particularly all of the six classification datasets used in Tables 1 and 13 . The results in Table 2 show that our method significantly outperforms CAAFE with GPT-4o, even when using our custom Llama 2 model fine-tuned on open dialogue data. Also, conventional feature engineering methods, such as OpenFE [17] , often struggle to generate meaningful features, particularly due to the difficulty of applying arithmetic operations to categorical features. A key distinction between CAAFE and OCTree is that our approach generates more semantically meaningful column names, which serve as a basis for creating high-quality features. Leveraging the LLM's reasoning and in-context learning capabilities, we guide the model in effectively navigating the feature space to generate coherent, relevant rules, using a history of feedback on candidate features and decision tree reasoning to enhance its understanding of the data. In contrast, CAAFE primarily relies on language understanding to suggest simple combinations of existing feature. Moreover, CAAFE tends to adopt a greedy approach, evaluating the validation score for a candidate feature only once and discarding it if no improvement is observed. We provide further results with case studies in Appendix E.Datasets without language descriptions. In practice, datasets do not always include clear language descriptions of the prediction task. For example, feature names and values in financial datasets are often obfuscated with arbitrary symbols to protect confidentiality [47] . OCTree adapts easily to such datasets without language descriptions and can generate features using various arithmetic rules. As shown in our ablations in Section 4.3, this is due to the more effective use of LLMs' optimization capabilities with decision tree reasoning that enhances the model's understanding of the data.To evaluate datasets without language descriptions, we apply ordinal encoding to categorical features and normalize all features using a min-max scaler, transforming the original numeric values. We also use non-descriptive column names, such as C = {'x1', 'x2', . . . , 'x5'} for a dataset with M = 5 columns. To initialize the feedback trajectory, we create an initial rule that is the product of the two columns with the highest importance weights computed using an XGBoost model, e.g.,x 6 = x 1 × x 5 .As shown in Table 3 , our framework enhances baseline models even in the absence of language descriptions, achieving an average error reduction of approximately 5.0% for both the XGBoost classifier and MLP. For HyperFast, OCTree also improves the test error on 16 out of 19 datasets. 4 , while all models yield improvements over the baseline, we find our own model (i.e., Llama 2 fine-tuned on UltraChat) to be particularly effective. This shows that our framework can be effectively implemented even with open models at a moderate scale, especially those with sufficiently strong chat capabilities. We suspect that this improvement stems from the enhanced ability of these models to understand and generate contextually relevant and coherent rules, resulting in better optimization outcomes. Specifically, as illustrated by the examples in Appendix F, our model navigates a broader space of features more effectively than the base Llama 2 chat model and demonstrates the ability to utilize built-in Python functions such as 'abs()'. Code Llama also demonstrates strong performance due to its training on code data that includes a variety of arithmetic rules, which is especially useful for generating rules for datasets lacking language descriptions. Notably, the model can leverage a range of NumPy operations (e.g., 'np.sin'), allowing it to produce more mathematically complex features. This suggests that LLMs trained on high-quality code and dialogue data could serve as even more effective rule generators within our framework.Integrating with other automated feature engineering methods. OCTree is complementary to existing feature engineering methods, allowing for integration in several natural ways. One simple approach is to first apply OCTree to generate features, followed by the use of other methods to further refine and augment the feature set. In Table 5 , we first compare the standalone performance of OCTree, demonstrating that it outperforms competing state-of-the-art automated feature engineering methods (i.e., AutoFeat [23] and OpenFE [17] ) for both XGBoost and MLP models. Moreover, combining OCTree with OpenFE (denoted as OCTree † in Table 5 ) further boosts performance, achieving a 7.9% reduction in relative error for XGBoost.Ablation study on the proposed components. As shown in Table 6 , our framework consists of two essential components: (i) generating new column features (denoted as Gen. Feat. in Table 6 ), and (ii) providing explicit decision tree reasoning as feedback (denoted as DT reasoning in Table 6 ) during the optimization process. First, note that the rules are sufficiently well-optimized even without an explicit decision tree provided as feedback; the LLM improves performance based solely on score feedback. However, providing the decision tree as feedback to the LLM can lead to even better performance. We believe that decision tree reasoning, which highlights important columns and their threshold values, enables the LLM to understand the data better, resulting in the generation of more contextually relevant and useful rules. Moreover, decision trees can be easily represented in natural language using if-else syntax, effectively conveying the information about the data to the LLM.Transferring generated rules to other prediction models. While we optimize feature generation rules to improve the performance of a specific prediction model, the generated features can also be utilized in other models to achieve similar improvements. For example, it would be more efficient to generate features using XGBoost, which typically trains and evaluates faster than larger deep neural networks, and then apply these features to more complex models. To assess whether such feature transfer is feasible within our framework, we first optimize the column generation rules using XGBoost and then train MLP and HyperFast models with the generated features. As shown in Table 7 , these features significantly improve the performance of both models, demonstrating the effectiveness of this approach, especially when only limited computational resources are available.Evaluating the validity of generated features. The rule generator LLM is asked to recommend a new column feature that is not already present in the dataset. Here, we evaluate whether the LLM is capable of generating features that are, in fact, valid and relevant to the target task. We perform this analysis from two perspectives: (i) whether the LLM can identify the most relevant column feature when provided with multiple candidate columns, and (ii) whether using real-world data, when available, for the suggested column leads to improved performance of prediction models.Our method is based on the assumption that sufficiently capable LLMs can understand the relationship between the target task and column features, enabling them to generate new, relevant features for the task. To evaluate this assumption, we first examine whether the LLMs can identify the features that are more relevant to the prediction task. For this experiment, we begin by removing two existing features from a dataset and then prompt the LLM to rank the two features according to their importance for the target task. Specifically, we remove 'Cholesterol Level' and 'Cough' from the original Disease dataset and then ask the LLM to identify the attribute more relevant to predicting whether a patient has a disease. Both the GPT-4o and Llama 2 models indicate that 'Cough' is more important than 'Cholesterol Level'. As shown in Table 8 , XGBoost achieves a lower error rate when trained with the 'Cough' feature compared to when trained with 'Cholesterol Level', which is consistent with the LLMs' assessment that the former is more relevant to the task. Building on the LLM's ability to identify important features for the target task, we further assess whether the generated columns can be imputed with real-world data to enhance prediction. In this experiment, we use the Clinical Trial dataset, where the LLM introduced the column 'Age' when prompted to suggest a new column. We then incorporated real-world age data from the US National Library of Medicine to augment the original dataset. As shown in Figure 3 , training on this augmented data results in a notable improvement, demonstrating that the LLM-generated columns align well with real-world data. In conclusion, we recommend that practitioners utilize OCTree to either (i) identify additional column features and collect the corresponding real-world data or (ii) optimize feature generation rules in the absence of real-world data.Analysis of the rule optimization process. We analyze how the output rules evolve throughout the optimization rounds on the electricity dataset. Due to space constraints, we show the first and last five output rules in Appendix H. In the early stages of optimization, the LLM generates a diverse range of outputs, indicating an active exploration of potential rules. In contrast, during the later stages, the LLM focuses on refining the solution space around previously identified rules, making only minor adjustments. This again demonstrates that with appropriate guidance through the optimization process, sufficiently capable LLMs can serve as highly effective optimizers.Handling hallucinations. While LLMs may occasionally suggest suboptimal or semantically incoherent rules, our method is designed to address these hallucinations. Specifically, we provide feedback on previously generated rules to guide the LLMs in iteratively improving the rule generation. This feedback loop helps the LLMs avoid hallucinations that might lead to low validation scores in subsequent iterations. Empirically, we find that these issues are more common in the early stages when the LLMs explore the rule space more broadly and in less capable models, e.g., those without additional training on dialogue or code generation data.A regression task to predict daily energy consumption based on the following attributes:  
conclusion: 
 In this paper, we propose OCTree, a generic framework that leverages the power of LLMs (e.g., reasoning capability) for automatically generating column features for tabular prediction tasks. We evaluate the effectiveness of OCTree across various prediction tasks and demonstrate that our method consistently enhances the performance of diverse prediction models, often significantly more effectively than competing feature engineering methods. As future work, exploring feedback-based alignment methods, such as reinforcement learning from human feedback, to further enhance LLMs as rule generators would be an exciting direction to explore.Limitation. One potential limitation of our work is that evaluating the generated features involves computing the validation scores of the prediction model, which can be time-consuming if the model requires extensive training. However, as demonstrated by the results in Table 7 , this issue can be mitigated by first generating features with a simpler prediction model and then transferring those features to the target model, reducing the overall runtime and computational requirements. Listing 2: Prompt for the rule initialization.A regression task to predict the target day's highest stock price based on the following attributes:• Numerical: Open Price of 2 Days Before, Highest Price of 2 Days Before, Lowest Price of 2 Days Before, Close Price of 2 Days Before, Open Price of 1 Day Before, Highest Price of 1 Day Before, Lowest Price of 1 Day Before, Close Price of 1 Day Before, Open Price of the Target Day, Time Index.1 https://www.kaggle.com/datasets/uom190346a/disease-symptoms-and-patient-profile-dataset 2 https://data.projectdatasphere.org/projectdatasphere/html/content/119 3 https://www.kaggle.com/datasets/missionjee/students-dropout-and-academic-success-dataset 4 https://www.kaggle.com/competitions/predict-energy-behavior-of-prosumers 5 https://www.kaggle.com/datasets/guillemservera/tsla-stock-data 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our framework does not introduce risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have cited all papers and datasets in Reference and Appendix B.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: In Listing 3, we present the prompt p gen , which instructs the LLM to generate an improved rule compared to those previously generated. The prompt includes a list of (rule, tree-based reasoning, score) tuples. Since the score represents the accuracy of the XGBoost classifier, the trajectory is sorted in ascending order of score. For regression tasks using mean absolute error, the trajectory is sorted in descending order. In Listing 4, we present the prompt p code , which instructs the LLM to translate the rule into Python code. To minimize potential syntactic errors, we impose several restrictions. First, we explicitly define the variable types wherever possible. We also instruct the LLM to account for the feature value types before performing calculations, e.g., to avoid the addition of categorical and numerical values. In this section, we provide further details on the datasets. A classification task to predict whether a student would dropout based on the following attributes:• Numerical: Marital Status, Daytime/Evening Attendance, Previous Qualification, Nationality, Father's Qualification, Father's Occupation, Displaced, Debtor, Tuition Fees up to Date, Gender, Scholarship Holder, Age at Enrollment, International, Curricular Units 1st Sem (Approved), Curricular Units 1st Sem (Grade), Curricular Units 2nd Sem (Approved), Curricular Units 2nd Sem (Grade).For our main results on datasets without language descriptions (see Table 3 ), we evaluate the 19 classification datasets from the tabular benchmark proposed by Grinsztajn et al. [13] . Following the curation approach from Grinsztajn et al. [13] ), we uniformly subsample to 50,000 instances for datasets exceeding this size. We provide brief dataset statistics below. Table 13 : Performance improvements by OCTree on datasets with language descriptions. We report test error rates (%) on six classification tasks ( * ) and mean absolute errors (×10 -3 ) for two regression tasks ( † ). The lowest error is in bold. Values in parentheses indicate the relative error rate reduction from the baseline. We report the mean error and standard deviation across three random splits, except for the two regression tasks (time series tabular data), which are split by time index. GPT-4o was used for both CAAFE and OCTree. We conduct a comparative evaluation using all datasets (along with additional ones) with contextual information from the experiments summarized in Table 1 . First, CAAFE [19] exhibited high variance, even on the same dataset split, partly due to the randomness associated with GPT-4o's temperaturebased sampling. At times, it failed to improve upon the baseline. To address this, we average the performance over three trials per random split and report the mean and variance. As shown in Table 13 , our method consistently outperforms CAAFE. Note the official implementation of CAAFE has been slightly modified to accommodate regression tasks. f''' If the individual has "Fever" and "Fatigue" and "Difficulty Breathing" with "Age" between 60 and 90, then predict "Exposure to Infected Individuals" as "Yes". Otherwise, predict "Exposure to Infected Individuals" as "No". As illustrated in Listings 5 and 6, CAAFE struggles to introduce meaningful columns for the disease dataset, whereas our method proposes more relevant and coherent rules.In Section 4.2, we evaluate three different LLMs and provide additional comparisons in this section.As shown in Finally, our custom model tends to explore a polynomial space of features while also utilizing built-in Python functions such as 'abs()'. Compared to the Llama 2 chat base model, it more effectively navigates a broader space of features:• x9 = x1 * * 2 * (x2 -x3) • x9 = abs(x1) * * x1 + x2 -x3 -x4 -x5 -x6 -x7 -x8 • x9 = x1 * (x1 + 2) -x2 * (x2 -0.5) * (x2 -0.5)These observations suggest that, while all three models can generate useful features, our custom model more effectively navigates a broader feature space, leading to the generation of even more valuable features. We suspect that further training on code data, particularly to enhance the ability to leverage scientific computing libraries like NumPy, could lead to additional performance improvements. Here, we evaluate the scalability of our method on datasets with hundreds of features (e.g., 501) from the OpenML repository [49] . We choose XGBoost [11] as the baseline model, because it is the most competitive baseline in our main experiments (see Table 3 ). Additionally, we use GPT-3.5-Turbo as the rule generator, as the Llama 2-based model we primarily use is constrained by a relatively short context length, which becomes limiting as the prompt size increases with the number of features. As shown in Table 14 , our method scales effectively to datasets with a larger number of features. For example, on the madelon dataset, which contains 501 columns, our method reduces the relative error by 6.3% compared to the XGBoost baseline.To identify an effective feature generation rule, we conduct 50 rounds of optimization with the LLM. In each round, the LLM is provided with an optimization trajectory that includes reasoning information from previous experiments. Consequently, the input prompt evolves throughout the optimization process, with later rounds incorporating more accumulated information. Below, we show the first and last five output rules generated during the optimization on the electricity dataset.First five:• x12 = x3 + (x5 * x8) * x2 -x6 • x12 = x5 * x2 + (x3 * x4 -x6) -x1 • x12 = x11 + (x7 * x10) * x2 -x8 • x12 = x3 + (x6 * x8 * x2) -x1 • x12 = (x1 * x6 + x2 * x8) * x3 -x7Last five:• x12 = ((x2 -x1) * (x11 -x4)) * * 2 -(0.5 * (x1 -0.3)) • x12 = ((x2 -x1) * (x11 -x4)) * * 6 • x12 = ((x2 -x1) * (x11 -x4)) * * 14 -(0.02 * (x5 -x7)) -0.5 • x12 = (x11 -x1) + (x11 * (x6 -x7)) • x12 = ((x2 -x1) * (x11 -x4)) * * 3 + (0.1 * (x5 -x6)) * * 2Note that in the early stages of optimization, the LLM tends to explore a wider variety of rules, allowing for greater exploration. In contrast, during the later stages, the model narrows its focus to refine features within a more constrained space.Our method is particularly effective in scenarios where collecting real data is costly or restricted, such as in the finance or medical domains, where data availability is often limited due to privacy concerns. However, since the features generated by OCTree are artificial, it is crucial to carefully inspect these features for their relevance and reliability. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We have discussed the societal impact in Appendix I. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: This authors propose a new tabular learning framework called OCTree (Optimizing Column Feature Generator with Decision Tree Reasoning) that leverages the reasoning capabilities of large language models (LLMs) to automatically generate new column features based on feedback from decision trees (i.e., it is an automatic feature space construction method). Methodologically, OCTree algorithm uses LLMs to recommend new features to be generated by name and feature generation rule, which is then used in the training of the ML algorithm for validation. Experimental results demonstrate that OCTree consistently outperforms existing automatic feature engineering methods across various tabular data prediction tasks. 

Consensus strengths of the paper identified by reviewers (and myself) include:
- The idea of using LLMs to generate novel features for ML models over tabular data is conceptually appealing and the specific approach proposed does this in a straightforward way (meant as a positive).
- The paper is very well-written, well-structured, and the empirical results are promising.
- The experiments have a sufficient number of LLMs, baseline algorithms, strong ML models for these tasks, benchmark datasets, and ablation studies to support the core statements claimed in the paper. I actually don't think the reviewers sufficiently appreciate how difficult XGBoost is to beat "out of the box" for these tasks -- which it is so widely used.

Additionally, reviewer concerns regarding comparison to CAAFE and scalability issues were sufficiently addressed during rebuttal.

Conversely, consensus (and my own) observed weaknesses of this work include:
- While issues regarding CAAFE and scalability were addressed during rebuttal, they still need to be integrated into the paper -- but this seems possibly just by editing a revised manuscript.
- Some experimental details (e.g., number of rounds, inter-round feedback) aren't easy to find in the paper (but are in the rebuttal). However, this is easily added.
- While the number of datasets is sufficient, the paper would be non-negligibly strengthened by more datasets, etc. as this would potentially show utility for different cases and could provides cases where interpretability, etc. are more salient (and in line with motivate). Honestly, this would make me more confident in recommending a clear accept.
- While I don't think that the existing experiments were 'polluted' by memorized features (i.e., reviewer 245r), this isn't too difficult to test for (at least for the Kaggle workshop cases) but could be a problem in competition settings -- but likely not an issue in actual deployed systems. Thus, I would include some discussion of this.
- In the same vein, i would add that the generated features are more interpretable than other feature space augmentation algorithms. Of course, generated features are often complex and thus formal interpretation is difficult, but the LLM labels and rules may be useful for this. Discussion is worthwhile as tabular data domains often have this aspect.
- I do agree with reivewer Pmbp that discussion regarding SDP-based feature generation methods is worthwhile; I don't believe empirical comparisons are needed, but this also occurred to me while reading.
- The discussion regarding hallucination in rebuttal should be addressed in the paper. It is clear to me that the ML algorithms can likely perform feature selection (as they have been doing for a long time), but should probably be mentioned.

Overall, I believe this is a nice paper that may have solid practical implications in many 'traditional ML' domains. Additionally, as LLMs continue to improve (including reasoning), I expect this to inspire future work. While there are some details that should be added to the paper, it is well-written overall and shows good performance and convincingly supports the core claims of the work.

Reviews: review: paper_summary: paper_summary: The authors introduce a novel feature engineering technique that leverages LLMs for language-based reasoning and considers the outcomes of past experiments as feedback for iterative rule improvements.
strength_weakness: strength_weakness: - the authors did not compare their method against other techniques for automated feature generation
- the authors provide no details on whether several rounds of LLM prompting were executed, how do results vary across multiple executions and whether they had issues with LLM hallucinations
questions: questions: We consider the paper interesting and relevant. Nevertheless, we would like to point to the following improvement opportunities:

GENERAL COMMENTS

(1) - The authors compare their model against models on which such feature generation was not applied, showing promising results. Nevertheless, we miss a comparison against similar methods that generate features either with heuristics or LLMs, as explained in the Related Work section. In particular, we would encourage the authors to include some comparison against the following work: Hollmann, Noah, Samuel Müller, and Frank Hutter. "Large language models for automated data science: Introducing caafe for context-aware automated feature engineering." Advances in Neural Information Processing Systems 36 (2024).

(2) - Did the authors execute several rounds of feature generation using the same prompt and input? Did they observe any variability regarding the results obtained?

(3) - How are results different when considering different LLMs? Are there certain patterns regarding the features generated/not generated? Furthermore, the authors mention that "incorporating code datasets alongside dialogue datasets may further enhance the performance of the rule generator LLM". How were such prompt results better from regular LLMs and different from those generated by the LLM proposed by the authors (e.g., in Table 3)?

(4) - Did the authors face LLM hallucinations? We would appreciate a brief description on how did they handle such cases and some quantification, to understand how frequent such problem is across different LLMs they tried. 


FIGURES

(6) - Figure 3: (i) can be made monochromatic, (ii) are datasets balanced? How is the cut threshold determined to measure Accuracy? Could this be replaced with a threshold-independent metric? 

TABLES

(7) - Table 7: bold the best results
limitations: limitations: The authors acknowledge some limitations of their work. Nevertheless, we consider some additional weaknesses (see weaknesses section) should be addressed and eventually acknowledged.
paper_summary: paper_summary: This paper proposes a new tabular learning framework called OCTree (Optimizing Column Feature Generator with Decision Tree Reasoning). The framework leverages the reasoning capabilities of large language models (LLMs) to automatically generate new column features based on feedback from decision trees. Experimental results demonstrate that this framework consistently outperforms existing automatic feature engineering methods across various tabular data prediction tasks.
The OCTree framework effectively utilizes the reasoning capabilities of LLMs and feedback from decision trees to automatically generate new column features, significantly enhancing the performance of tabular data prediction tasks. This framework demonstrates the great potential of LLMs in automatic feature engineering for tabular data.
strength_weakness: strength_weakness: w1: I thoroughly enjoyed reading this work. The authors present a unique perspective on the optimization of tabular data tasks. For example, in the abstract, the authors highlight the shortcomings of existing work, such as "they often rely solely on validation scores to select good features, neglecting valuable feedback from past experiments that could inform the planning of future experiments." This indeed reflects the current issues faced by Automated Feature Transformation, where high-performing features may not necessarily be meaningful features. However, conversely, it seems that the goal of Feature Engineering is to improve the performance of downstream machine learning tasks, which is a very direct objective. Isn't it contrary to the original intent of feature engineering if validation scores are not used as an evaluation metric?

w2: The writing of the entire paper is relatively easy to read, although some concepts were not discussed in depth. For instance, the authors only claim that validation scores are a flawed feedback mechanism but do not explore some of the newer methods in the main text, such as the latest approaches utilizing reinforcement learning [1-2] or generative artificial intelligence [3-4]. These methods employ the performance of models like decision trees and other machine learning models as scores to learn strategies.

w3: In Figure 1, Fever, Fatigue, Breathing, etc., are used through processes like Tabular Data Understanding and rule generation to combine boolean forms (Fever, Fatigue) or one-hot forms of Breathing to create a new feature, Smoke. This is quite intuitive; however, if the rules generated in step 1 are inaccurate, the generated column might also be inaccurate. Moreover, can the concept of whether one smokes or not be automatically and adaptively formed during the generation of decision trees or the training of neural networks, rather than requiring such an inaccurate combination? If not, do the authors have experiments or specific mathematical proofs to validate this claim or to show that neural networks, for instance, would converge more difficultly without these generated key features?

w4: In the Column name generation section of the Methodology (section 3.2), how do the authors ensure that the generated column names are valid?

w5: In Table 2, it seems that some datasets with non-linguistic descriptions did not improve or only showed general improvement. What are the specific reasons for this?

Although this work seems overly intuitive at times, I remain excited about the application of large language models in this field. I hope the authors can address the above questions.

[1] Traceable group-wise self-optimizing feature transformation learning: A dual optimization perspective, TKDD
[2] Group-wise Reinforcement Feature Generation for Optimal and Explainable Representation Space Reconstruction, KDD
[3] Reinforcement-Enhanced Autoregressive Feature Transformation: Gradient-steered Search in Continuous Space for Postfix Expressions, NeurIPS
[4] DIFER: Differentiable Automated Feature Engineering, ICAML
questions: questions: see weakness
limitations: limitations: see weakness
paper_summary: paper_summary: The authors propose an automatic feature engineering method called OCTree. The OCTree algorithm uses LLMs to generate new features. The feature is then used in the training of the black box, and the new validation score is stored. The algorithm uses a decision tree to learn rules that show the reasoning behind selecting these new features and incorporate this in an iterative process. 


NOTE: If the authors address the issues I have raised in limitations and answer my questions, I will improve my score.
strength_weakness: strength_weakness: It is very difficult for me to follow the notation and understand what the algorithm is doing. This gets harder as the authors have not released the code for the experiments. 

The authors keep iterating that the main limitation of another feature engineering is that they require setting the manual search space without showing how their approach is solving this limitation.

The experiments do not include CAAFE, the most relevant algorithm to OCTree. The authors claim that the reason behind this is that CAAFE relies on a language-based context of the dataset, making it inapplicable to the datasets they have used. But I am not yet convinced. 

None of the decision-tree reasoning and how it is improving OCTree is explained in the paper.
questions: questions: Why have the authors excluded CAAFE? I looked at the tutorials, and they seem very easy to use. 

Why do the authors say that CAAFE is orthogonal to their approach? I do not think that is the case. I understand that your approach is CAAFE + a decision tree process embedded. Can you elaborate on the difference between your approach and CAAFE with a concrete example?  

Your notations are so confusing. In Line 131, what does D + r mean? D is a dataset, and r is a rule; they cannot be summed up. 

Can authors present evidence on how the inclusion of rules learned by decision trees is increasing the efficiency of their algorithm?
limitations: limitations: The main competitive algorithm in this work, CAAFE, is excluded from the study. Therefore, we do not have a good baseline for how this approach can improve the problem.   

The authors have not addressed the study's main concern: the risk of memorization. LLMs are possibly trained on the open datasets they are using. This could be that the LLM is, for example, summarizing what most Kaggle blogs and Medium posts on these datasets have done.
paper_summary: paper_summary: The paper introduces OCTree, a framework that uses LLMs to generate and refine feature generation rules for tabular data. By incorporating decision tree reasoning, OCTree iteratively improves feature generation using feedback from past experiments. The framework enhances the performance of some prediction models, including decision trees and neural networks, and demonstrates improvements on real-world datasets. OCTree works with datasets with or without language descriptions and outperforms existing feature engineering methods, showing flexibility in feature generation for prediction tasks.
strength_weakness: strength_weakness: 1. The framework is costly and time-consuming due to the need for training new models for validation during each iteration, and it involves some manual operations, preventing full automation.
2. Lack of comparisons of training times makes it difficult to assess the efficiency relative to other approaches.
3. The experiments are limited to datasets with a maximum of 54 features, lacking evidence of effectiveness on high-dimensional data (e.g., datasets with hundreds of features).
4. Datasets are sampled to sizes smaller than 50,000 instances, raising concerns about effectiveness and scalability on larger datasets.
questions: questions: 1. Are there other works that use LLMs for feature generation? If so, how does this approach differ from them?
2. What is the time cost associated with OCTree? How much slower is it compared to directly training XGBoost? Does the time cost increase significantly with the size of the dataset and the feature dimensionality?
limitations: limitations: N/A

Comments: title: Thank you for the additional feedback
Thank you very much for the response
Thank you for the additional feedback (2/2)
Thank you for the additional feedback (1/2)
Thank you very much for the response
comment: Dear Reviewer 8DXj,

Thank you for the additional feedback. Here, we would like to further address your concerns individually.

---
**For [W1, 2], although this paper focuses on the method's effectiveness, the improvement over the baseline in many datasets is not substantial (for example, in Table 2, where the comparison with XGBoost shows that in 19 datasets, the relative error rate reduction is less than 4% in 11 datasets). Therefore, I believe that a training time comparison is necessary.**

Thank you for your feedback. We understand the importance of evaluating training time. The run time of our method depends on the number of additional features generated and the number of optimization steps used to refine each feature. As you noted, these factors can be substantial depending on dataset size and model type.

Nevertheless, we would like to highlight that our method demonstrates consistent improvement in performance across (i) datasets and (ii) baseline model types. As shown in Table 4 of our manuscript, our method outperforms competing automatic feature engineering methods, including AutoFeat and OpenFE. In particular, OpenFE fails to introduce useful features for multilayer perceptron (MLP) in 9 of the 22 classification datasets used in Tables 1 and 2, which contrasts with our method's more robust performance.

In response to your comment, we will assess potential speed-ups through approaches such as feature transfer and reducing training data during feature generation that we highlighted in our previous response. We will include these evaluations and additional quantitative results on training time in the final revision.

---
**For [W4], the main issue is not whether the method performs well on one or two datasets with more than 50,000 instances, but rather that the paper's experimental section should avoid sampling datasets down to fewer than 50,000 instances, as this is likely to cause the baselines to underperform.**

Thank you for highlighting this concern. We sampled datasets down to fewer than 50,000 instances to align with the experimental setup used in [1] (see Appendix A.2.2), which provides a comprehensive benchmarking study of tree-based and deep learning methods on tabular data. This alignment facilitates a clearer comparison of our results with those from previous works.

We also want to clarify that once a dataset is sampled, the same sampled dataset is used for both the baselines and our method. Thus, while sampling may impact baseline performance, as you pointed out, our feature generation method is subject to the same conditions. Additionally, it's relevant to note the experimental setup of the previous automatic feature engineering method, CAAFE [2], which only focuses on small datasets with up to 2,000 samples.

In contrast, our method has been evaluated on datasets with significantly larger feature dimensions and sample sizes, as detailed in the additional experiments provided in our previous response. We hope these results address your original concern and we will ensure that these results are clearly incorporated into the final revision to comprehensively address your concerns.

[1] Grinsztajn et al., Why do tree-based models still outperform deep learning on tabular data? NeurIPS 2022.\
[2] Hollman et al., LLMs for Semi-Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering, NeurIPS 2023.

---
**For [Q1], since there are related comparative methods, a comparison between this method and those methods should be included in all evaluation parts of the paper, e.g., Table 2.**

Thank you for your feedback. We would like to clarify that we have compared our approach to CAAFE on **all** datasets where CAAFE is applicable, as detailed in our **global response** and the **attached PDF**. Specifically, we performed a comparative evaluation using all datasets that include contextual information, as shown in Table 1 of our manuscript. Since CAAFE requires language-based contexts, which are not available for all datasets (e.g., those used in Table 2), our comparison focused on datasets where CAAFE can be effectively applied. This highlights the broader applicability of our method compared to CAAFE. We hope this explanation clarifies our approach and the scope of our comparisons.
I appreciate the author's efforts in addressing the questions and concerns. However, there are still some critical issues that remain unresolved.

1. For [W1, 2], although this paper focuses on the method's effectiveness, the improvement over the baseline in many datasets is not substantial (for example, in Table 2, where the comparison with XGBoost shows that in 19 datasets, the relative error rate reduction is less than 4% in 11 datasets). Therefore, I believe that a training time comparison is necessary.
2. For [W4], the main issue is not whether the method performs well on one or two datasets with more than 50,000 instances, but rather that the paper's experimental section should avoid sampling datasets down to fewer than 50,000 instances, as this is likely to cause the baselines to underperform.
3. For [Q1], since there are related comparative methods, a comparison between this method and those methods should be included in all evaluation parts of the paper, e.g., Table 2.
Dear reviewer 6JM8,

Thank you very much for letting us know! We are happy to hear that our rebuttal addressed your questions well.\
If you have any further questions or suggestions, please do not hesitate to let us know.

Thank you very much,\
Authors
We appreciate the authors' effort in answering our questions and the other reviewers' questions. We have no further questions and have decided to update our scores and slightly increase the final score.
**[F4] Yeah, but you cannot really say that those features do not come from a tutorial on feature engineering on Kaggle in 2015, right? That is what I meant.**

Thank you for your insightful comments. We understand your concern that LLMs may have memorized feature engineering techniques, such as those from the tutorials on Kaggle. Indeed, our method leverages LLM’s ability to introduce engineering rules from existing features. However, while LLMs may be familiar with established techniques, this does not guarantee that they can suggest effective feature engineering rules for every dataset. For example, without contextual information about a column, an LLM may struggle to identify which features are most relevant and, therefore, may not provide useful rules.

Our method addresses this limitation by proposing an optimization process to find good feature generation rules by designing novel and effective feedback mechanisms. Specifically, in the early optimization stages, the LLM explores a wide range of potential rules, which may be based on common engineering techniques learned, as you pointed out. As the process continues, the LLM refines the solution space around previously discovered solutions, making only minor adjustments during the later stages. In short, our method goes beyond simply utilizing the LLM’s existing knowledge of feature engineering techniques by actively guiding it toward discovering the most effective rules for the specific dataset.

We appreciate your feedback and will incorporate this discussion into the final draft.
**[F1] Absolutely. That's the point of peer review. However, I would like to say that writing does not necessarily mean that the notations are accurate. It can be seen that the "story" is well written.**

We appreciate the constructive feedback and agree that clarity in notation is crucial. As we noted in our earlier response, we recognize that certain notations (e.g., $D \oplus r$) could be made clearer. We will carefully review and refine these notations in the final draft to ensure ease of understanding.

---
**[F2] Sure, but then you must also compare CAAFE to your approach in cases where it is possible. Or is your approach ONLY applicable to those cases? In that case, I would say your solution is very niche.**

We would like to clarify that our method is applicable to both cases where language descriptions are clearly provided and where they are not (see Tables 1 and 2 in our manuscript). We fully agree that comparing our approach with CAAFE is important. Therefore, as per your suggestion, we have already compared CAAFE to our approach in the previous rebuttal using all datasets with contextual information from the experiments summarized in Table 1 of our manuscript.

To summarize the results, Table 1 of the attached PDF shows that our method consistently outperforms CAAFE. For a more detailed discussion, we kindly refer you to the **global response** and the **attached PDF**. We appreciate your suggestion and will incorporate these comparative results in the final draft.

---
**[F3] I appreciate it, but what I mean is that we need to see the decision tree rules and how the tree relates to the process.**

Thank you for your question and the opportunity to provide further clarification. First of all, we would like to clarify that decision tree reasoning refers to the language description of a decision tree that is constructed from the entire dataset, including any newly generated features. This reasoning is then used to guide the LLM in generating more effective feature generation rules.

For instance, consider the following example:
- Original columns: Fever, Breathing
- Introduced column: Smoking status
- Task: Predicting whether a patient has a disease

A decision tree derived from this data might produce reasoning such as:

```
if ‘Has difficulty breathing’:
     if ‘Has fever’:
          ‘Subject has a disease’
     else:
          ‘No disease’
…
```

This decision tree reasoning is provided to the LLM as feedback to refine its feature generation process. Here, we provide a prompt that is used to guide LLM to find good feature generation rules (see Appendix A.3).

```
I have some rules to predict {Objective} with attributes listed below.
- {Column #1 Name} (Numerical value of {Column #1 Min} ~ {Column #1 Max}):  {Column #1 Feature Description}
- {Column #2 Name} (Boolean):  {Column #2 Feature Description}
- {Column #3 Name} (Categorical value of {Value #1}, {Value #2}, {Value #3}):  {Column #3 Feature Description}
...

We also have corresponding decision trees (CART) to predict {Objective} from the attributes listed above along with predicted {New Column Name}.
The rules are arranged in ascending order based on their scores evaluated with XGBoost classifier, where higher scores indicate better quality.

Rule to predict {Objective}:
{Rule #1}
Decision tree (CART):
{Decision Tree #1}
Score evaluated with XGBoost classifier:
{Score #1}

Rule to predict {Objective}:
{Rule #2}
Decision tree (CART):
{Decision Tree #2}
Score evaluated with XGBoost classifier:
{Score #2}

...

Give me a new rule to predict {Objective} that is different from the old ones (but should use the listed attributes above) and has a score as high as possible.

Improved rule:
```
We appreciate the question and will provide more examples and explanations in the final draft.
I thank the authors for answering my concerns. I appreciate it. Unfortunately, I am not convinced after their answers. 

Some more feedback is below:  

> While several reviewers have noted that the paper is generally well-structured and written

Absolutely. That's the point of peer review. However, I would like to say that writing does not necessarily mean that the notations are accurate. It can be seen that the "story" is well written. 

> This requirement limits CAAFE's applicability in cases where language descriptions are not explicitly provided, such as when feature names and values are obfuscated to protect confidentiality, as is common in financial and medical datasets.

Sure, but then you must also compare CAAFE to your approach in cases where it is possible. Or is your approach ONLY applicable to those cases? In that case, I would say your solution is very niche. 

>We have already conducted an ablation study, detailed in Table 5, to assess the effects of providing decision tree reasoning as feedback to the rule generator LLM. 

I appreciate it, but what I mean is that we need to see the decision tree rules and how the tree relates to the process. 

>Even if the LLM has memorized information from open datasets, our approach does not rely on the specific data; instead, it generates and introduces entirely new columns to the dataset.

Yeah, but you cannot really say that those features do not come from a tutorial on feature engineering on Kaggle in 2015, right? That is what I meant.
Dear reviewer Pmbp,

Thank you very much for letting us know! We are happy to hear that our rebuttal addressed your questions well.\
Also we thank you for your prompt response.\
If you have any further questions or suggestions, please do not hesitate to let us know.

Thank you very much,\
Authors
I appreciate the authors' effort in addressing my questions in their response. As reflected in my scores，my overall assessment is positive, and I maintain my current scores.