Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering
Authors: Noah Hollmann
Samuel Müller
Frank Hutter
Venue: NeurIPS 2023 poster (2023)
Keywords: AutoML
AutoDS
Automated Feature Engineering
LLM Code Generation
Tabular Data
Feature Engineering
Automated Data Science
Automated Machine Learning
TL;DR: We introduce CAAFE which automatically creates interpretable and semantically meaningful features and paves the way for more extensive (semi-)automation in data science tasks.

Abstract: As the field of automated machine learning (AutoML) advances, it becomes increasingly important to incorporate domain knowledge into these systems.
We present an approach for doing so by harnessing the power of large language models (LLMs). Specifically, we introduce Context-Aware Automated Feature Engineering (CAAFE), a feature engineering method for tabular datasets that utilizes an LLM to iteratively generate additional semantically meaningful features for tabular datasets based on the description of the dataset. The method produces both Python code for creating new features and explanations for the utility of the generated features.

Despite being methodologically simple, CAAFE improves performance on 11 out of 14 datasets -- boosting mean ROC AUC performance from 0.798 to 0.822 across all dataset - similar to the improvement achieved by using a random forest instead of logistic regression on our datasets. Furthermore, CAAFE is interpretable by providing a textual explanation for each generated feature.

CAAFE paves the way for more extensive semi-automation in data science tasks and emphasizes the significance of context-aware solutions that can extend the scope of AutoML systems to semantic AutoML. We release our [code](url{https://github.com/automl/CAAFE), a simple [demo](url{https://colab.research.google.com/drive/1mCA8xOAJZ4MaB_alZvyARTMjhl6RZf0a) and a [python package](url{https://pypi.org/project/caafe/).

Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering
Noah Hollmann
Samuel Müller
Frank Hutter
University Freiburg
Charité Hospital Berlin
University of Freiburg Prior Labs muellesa@cs uni meiifreiburg.co de
University of Freiburg Prior Labs
Prior Labs noah. Nooh.homa@gaalll com
fhècs. uni mai-fre ceii-frebburgg de
Abstract
As the field of automated machine learning (AutoML) advances, becomes in- creasingly important to incorporate domain knowledge into these systems. We present an approach for doing so by harnessing the power large language models (LLMs). Specifically, we introduce Context-Aware Automated Feature Engineering (CAAFE), feature engineering method for tabular datasets that utilizes an LLM iteratively generate additional semantically meaningful features for tabular datasets based on the description of the dataset. The method produces both Python code for creating new features and explanations for the utility of the generated features.
Despite being methodologically simple, CAAFE improves performance 11 out 14 datasets boosting mean ROC AUC performance from 0.798 to 0.822 across all dataset similar the improvement achieved by using random forest instead of logistic regression on our datasets.
Furthermore, CAAFE is interpretable by providing textual explanation for each generated feature. CAAFE paves the way for more extensive semi-automation in data science tasks and emphasizes the significance of context-aware solutions that can extend the scope of AutoML systems to semantic AutoML. We release our code, simple demo and python package.
CAAFE
Evaluate Performance.
Keep change if performance is improved.
Tabular Pre- diction Model: Performs
User: Specifies problem context and dataset
LLM: Generates
Interpreter: Executes generated code
Code for feature
engineering
cross-validation.
Figure 1: CAAFE accepts dataset as well as user-specified context information and operates by iteratively proposing and evaluating feature engineering operations.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
Automated machine learning (AutoML; e.g., Hutter et al. (2019)) very effective at optimizing the machine learning (ML) part of the data science workflow, but existing systems leave tasks such as data engineering and integration of domain knowledge largely to human practitioners. However, model selection, training, and scoring only account for small percentage of the time spent by data Science"(Anaconii
scientists (roughly 23% according to the "State of Data Science" (Anaconda. 2020)). Thus, the most
time-consuming tasks, namely data engineering and data cleaning, only supported to a very limited degree by AutoML tools, if at all.
While the traditional AutoMI. approach has been fruitful and appropriate given the technical capabili- ties of ML tools the time, large language models (LLMs) may extend the reach of AutoMI. to cover more of data science and allow it to evolve towards automated data science (De Bie et al., 2022). LLMs encapsulate extensive domain knowledge that can be used to automate various data science tasks, including those that require contextual information. They are, however, not interpretable, or verifiable, and behave less consistently than classical ML algorithms. E.g., even the best LLMs still fail to count or perform simple calculations that are easily solved by classical methods (Hendrycks et al., 2021; ;; OpenAI Community, 2021).
In this work, we propose an approach that combines the scalability and robustness of classical ML classifiers (e.g. random forests (Breiman, 2001)) with the vast domain knowledge embedded in LLMs, as visualized in Figure 2. We bridge the gap between LLMs and classical algorithms by using code as an interface between them: LLMS generate code that modifies input datasets, these modified datasets can then be processed by classical algorithms. Our proposed method, CAAFE, generates Python code that creates semantically meaningful features that improve the performance of downstream prediction tasks in an iterative fashion and with algorithmic feedback as shown in Figure Furthermore, CAAFE generates comment for each feature which explains the utility of generated feature. This allows interpretable AutoML, making it casier for the user to understand a solution, but also to modify and improve on it. Our approach combines the advantages of classical ML (robustness, predictability and level of interpretability) and LLMs (domain- knowledge and creativity).
Automating the integration of extend the scope of existing AutoML. methods. These benefits include: i) Reducing the latency from data to trained models; ii) Reducing the cost of creating ML models; iii) Evaluating a more informed space of solutions than previously possible with AutoML, but a larger space than previously possible with manual approaches for integrating domain knowledge; and iv) Enhancing the robustness and reproducibility of solutions, as sepputer--geeratedd solutions are more easily reproduced. CAAFE demonstrates the potential of LLMs for automating a broader range of data science tasks and highlights the emerging potential for creating more robust and context-aware AutoML tools.
into the AutoML process has clear advantages that
2 Background
2.1 Large Language Models (LLMs)
LLMs are neural networks that are pre-trained on large quantities of raw text data to predict the next word in text documents. Recently, GPT-4 has been released as a powerful and publicly available LLM (OpenAl, 2023a). The architecture of GPT-4 is not oublished, it is likely based on a deep neural network that uses transformer architecture (Vaswani al., 2017). large-scale pre-training diverse corpus of text and fine-tuning using reinforcement learning from human feedback (RLHF) (Ziegler al., 2019). It achieves state-of-the-an art performance summarization, question answering and coding. One can adapt LLMs to a specific task without retraining by writing a prompt (Brown et al., 2020; Wei et al., 2021); ); the model parameters are frozen and the model performs in-context inference tasks based on a textual input that formulates the task and potentially contains examples.
various tasks, such text generation,
LLMs as Tabular Prediction Models Hegselmann et al. (2023) recently showed how to use LLMs for tabular data prediction by applying them to a textual representation of these datasets. A prediction on an unseen sample then involves continuing the textual description of that sample on the target column. However, this method requires encoding the entire training dataset as a string and processing it using transformer-base architecture, where the computational cost increases
2

---

CAAFE
Traditional AutoML
Context specification contextual informa- tion, goal analysis and specification
Exploitation interpretation, visual- ization, reporting, pre- dictions, monitoring
Data Engineering wrangling, integra- tion, preparation, transfrmattion
Model Building algorithm choice, parameter ing. evaluation, model selection
Strength of LLMs:
Strength of classical algorithms: Well-defined, predictable
User driven
Exploratory, context deeexdtdeeedeent, real-world knowledge
Figure 2: Data Science pipeline, inspired by De Bie et al. (2022). CAAFE allows for automation of semantic data engineering, while LLMs could provide even further automation: (1) Context specification is user driven (2) exploitation and data engineering be automated through LLMs (3) model building can be automated by classical AutoML approaches.
quadratically with respect N M, where N denotes the number samples and M the number of features. Furthermore, the predictions generated by LLMs are not easily interpretable, and there is no assurance that the LLMs will produce consistent predictions, as these predictions depend directly on the complex and heterogencous data used to train the models. So far, Hegselmann et al. (2023) found that their method yielded the best performance on tiny datasets with up to 8 samples, was outperformed for larger data sets.
LLMs for Data Wrangling Narayan et al. (2022) demonstrated tiee-offtheeatr results using LLMs for entity matching, error detection, and data imputation using prompting and manually tuning the LLMs. Vos et al. (2022) extended this technique by employing an improved prefix tuning technique. Both approaches generate and utilize the LLMs output for each individual data sample, executing prompt for each row. This is in contrast to CAAFE, which uses code an interface, making our work much more scalable and faster to execute, since one LLM query can be applied to all samples.
2.2 Feature Engineering
Feature engineering refers to the process of constructing suitable features from raw input data, which can lead to improved predictive performance. Given dataset D (zz, y0) the goal is to find function é A. Common methods include numerical transformations, categorical encoding, clustering, group aggregation, and dimensionality reduction techniques, such as principal component analysis (Wold al., 1987).
X which maximizes the performance of A(p(x;), y1) for some learning algorithm
Deep learning methods are capable of learning suitable transformations from the raw input data making them more data-driven and making explicit feature engineering less critical, but only given a data. Thus, appropriate feature engineering still improves the performance of classical and deep learning models, particularly for limited data, complex patterns, or model interpretability.
Various strategies for automated feature engineering have been explored in prior studies. Deep Feature Synthesis (DFS; Kanter & Veeramachaneni (2015)) integrates multiple tables for feature engincering by enumerating potential transformations on features and performing feature selection based on model performance. Cognito (Khurana et al., 2016) proposes tree-like exploration of the feature space using handcrafted heuristic traversal strategies. AutoFeat (Horn al., 2019) employs an iterative subsampling of features using beam search. Learning Learning-based methods, such as LFE (Nargesian al., 2017), utilize machine learning models to recommend beneficial transformations while other methods use reinforcement Barning-based strategies (Khurana et al., 201 8; Zhang et al., 2019). Despite these advancements, none of the existing methods can harness semantic information in automated manner.

---

2.2.1
Incorporating Semantic Information
Initial data
With extra feature based on context
Weekend
Weekday
2
4 6 8 9 1011121114
2
3
4
5
6 7 9 12111
True False o Test-Sample
Working Day-Off o Test-Sample
Figure 3: Contextual information can simplify a task immensely. On the left-hand side no contextual information is added to the plot, and it is hard to predict the label for the green query point. On the right-hand side contextual information is added and a useful additional feature (weekend or weekday) is derived from which mapping from features to targets can be found.
The potential feature space, when considering the combinatorial number of transformations and combinations, is vast. Therefore, semantic information is useful, to serve prior for identifying useful features. By incorporating semantic and contextual information, feature engineering techniques can be limited to semantically meaningful features enhancing the performance by mitigating issues with multiple testing and computational complexity and boosting the interpretability of machine learning models. This strategy is naturally applied by human experts who leverage their domain- specific knowledge and insights. Figure 3 exemplifies the usefulness of contextual information.
3 Method
We present CAAFE, an approach that leverages large language models to incorporate domain knowledge into the feature engineering process, offering promising direction for automating data science tasks while maintaining interpretability and performance.
Our method takes the training and validation datasets, Dtrain and Doolid, well description of the context of the training dataset and features as input. From this information CAAFE constructs prompt, i.e. instructions to the LLM containing specifics of the dataset and the feature engineering task. Our method performs multiple iterations of feature alterations and evaluations on the validation dataset, outlined in Figure 1. In each iteration, the LLM generates code, which is then executed on the current D Dtrain and D Dealid resulting in the transformed datasets D troin and D' valid- We then D'rain to fit MI-classifier and evaluate its performance P' Dratid- If Pd exceeds the performance achieved by training on Dirain and evaluating on Dvalid, the feature is kept and set Dtrain D D'rain and Dvalid D Deatid- Otherwise, the feature is rejected and Dtrain and D Dvalid remain unchanged. Figure 4 shows a shortened version of one such run on the Tic-Tac-Toe Endgame dataset.
Prompting LLMs for Feature Engineering Code Here, we describe how CAAFE builds the prompt that used perform feature engineering. In this prompt, the L.LM instructed create valuable features for a subsequent prediction task and to provide justifications for the added feature's s utility. It also instructed drop unnecessary features, e.g. when their information captured by other created features.
The prompt contains semantic and descriptive information about the dataset. Descriptive information, i.c. summary statistics, such as the percentage of missing values is based solely on the train split of the dataset. The prompt consists of the following data points:
A A user-generated dataset description, that contains contextual information about the dataset (see Section 4 for details on dataset descriptions for our experiments)
Feature names adding contextual information and allowing the LLM to generate code to index features by their names

---

Dataset description: Tic Tac Toe Endgame database This database encodes the complete set of possible board configurations at the end of tic-tac-tos games, where "x" is assumed to have played first. The target concept is "win for x" (i .e., true when has one of possible ways to create a "three- in-a-rov").
-f -x-iins', Rumber of Usefulness: Knowing the number of ways x can vin on the board can be useful in predicting whether has
win the board')
the
Input samples: 'tap-left hop-left-ar letae-etaaar"" [2, 2, 1J, "top-middle sadidlle-quuree"" [t, 2, 01 ((df (ttpp--lltt-equureeee-- dr ('tap-right- (qquree]===] asttpe + ((df ('middle soaddddlllffttte square//urr))== (dr (middde (dr ['middle rright righht-auurre -aquare (]==1))
(dr["top- square (ar
-Middle-square
aatype (int) [...]
Iteration Performance before adding features ROC 0 888, ACC 0. 70000.
Performance after adding features ROC 0. 0.987, ACC 980.
Improvement ROC 0. 099 ACC 0. 08280. Code executed and changes to df retained.
C'uubber off -o-vina?, "Number of ways o can win on the board') Usefulness: Xoowing
ways
board
useful
predicting whether o has ven the game or not.
Input samples: hap-left--quazee
[2, 2, 10, 'top-mii "top-middlo midddidddd-lo-- square" [1, 2, a (ar squtrss]uu]r]]==])
df ['nunber -o -o--o--ins
((dr ('top Siiptrright-oe Siight--quarre]B square')-=2) astype ((df ('middle -left square/]==22 (df ('midde (df ['middee aquare (]==)) astype (int) [...]
-middle- Saddle-squarae330
Iteration
Performance before adding features ROC 0. 987, ACC 0. 0.980.
Performance after adding features ROC 1. 1.000, ACC 1.000.
Improvement ROC 0. 0.013, ACC 0.020. Code was executed and changes to df retained.
Figure 4: Exemplary of CAAFE the Tic-Tac-Toe Endgame dataset. User generated input is shown in blue, ML-classifier generated data shown in red and LLM generated code is shown with syntax highlighting. The generated code contains comment per generated feature that follows a template provided in our prompt (Feature name, description of usefulness, features used in the generated code and sample values of these features). In this run, CAAFE improves the ROC AUC on the validation dataset from 0.888 to 1.0 in two feature engineering iterations.
C Data types (c.g. float, int, category, string) this adds information how to handle feature in the generated code
D Percentage of missing values missing values additional challenge for code generation
10 random rows from the dataset - this provides information on the feature scale, encoding, etc.
Additionally, the prompt provides a template for the expected form of the generated code and explanations. Adding template when prompting is common technique to improve the quality of responses (OpenAI, 2023b). We use Chain-of-thouuht instructions instructing a series of intermediate reasoning steps -, another effective technique for prompting (Wei et al., 2023). The prompt includes an example of one such Chain-of-thought for the code generation of one feature: first providing the high-level meaning and usefulness of the generated feature, providing the names of features used to generate it, retrieving sample values it would need accept and finally writing a line of code. We provide the complete prompt in Figure 5 in the appendix.

---

If the execution of code block raises an error, this error is passed to the LLM for the next code generation iteration. We observe that using this technique CAAFE recovered from all errors in our experiments. One such example can be found in Table 3.
Technical Setup The data is stored in a Pandas dataframe (Wes McKinney, 2010), which is preloaded into memory for code execution. The generated Python code is executed in an environment where the training and validation data frame is preloaded. The performance is measured on the current dataset with ten random validation splits Dallid and the respective transformed datasets D Diali with the mean change of accuracy and ROC AUC used to determine if the changes of a code block are kept, i.e. when the average of both is greater than O. We use OpenAl's GPT-4 and GPT-3.5 as LLMs (OpenAI, 2023a) in CAAFE. We perform ten feature engineering iterations and TabPEN (Hollmann et al., 2022) in the iterative evaluation of code blocks.
The automatic execution of Al-generated code carries inherent risks, such as misuse by malicious actors unintended consequences from AI systems operating outside of controlled environments. Our approach is informed by previous studies on Al code generation and cybersecurity (Rohlf, 2023; Crockett, 2023). We parse the syntax of the generated python code and use a whitelist of operations that are allowed for execution. Thus operations such as imports, arbitrary function calls and others are excluded. This does not provide full security, however, e.g. does not exclude operations that can lead to infinite loops and excessive resource usage such as loops and list comprehensions.
4 Experimental Setup
Setup of Sawaasttram-CClasiiie We evaluate method with Logistic Regression, Random Forests (Breiman, 2001) and TabPFN (Hollmann et al., 2022) for the final evaluation while using TabPFN to evaluate the performance of added features. We impute missing values with the mean, one-hot or ordinal encoded categorical inputs, normalized features and passed categorical feature indicators, where necessary, using the setup of Hollmann et al. (2022)
Setup of Automated Feature Engineering Methods We also evaluate popular context-agnostic feature engineering libraries Deep Feature Synthesis (DFS; Kanter & Veeramachaneni, 2015) and AutoFeat (Horm et al., 2019)². We evaluate DFS and AutoFeat alone and in combination with CAAFE. When combined, CAAFE is applied first and the context-agnostic AutoFE method subsequently. For DFS we use the primitives For TabPFN, DFS generates more features than TabPFN accepts (the maximum number of features is 100) in some cases. In these cases, we set the performance to the performance without feature engineering. For AutoFeat, we use one feature engineering step and default settings otherwise.
didddd_" and "multiply_numerrc"" and default settings otherwise.
Evaluating LLMS on Tabular Data The LLM's training data originates from the web, potentially including datasets and related notebooks. GPT-4 and GPT-3.5 have a knowledge cutoff in September 2021, i.e., almost all of its training data originated from before this date. Thus, an evaluation on established benchmarks can be biased since textual description of these benchmarks might have been used in the training of the LLM.
We use two categories of datasets for our evaluation: (1) widely recognized datasets from MLenML released before September 2021, that could potentially be part of the LLMS training corpus and (2) lesser known datasets from Kaggle released after September 2021 and only accessible after accepting an agreement and thus harder to access by web crawlers.
From OpenML (Vanschoren et al., 2013; Feurer et al.), we use small datasets that have descriptive feature names (i.e. we do not include any datasets with numbered feature names). Datasets on OpenML contain task description that we provide as user context to our method. When datasets are perfectly solvable with TabPFN alone (i.e. reaches ROC AUC of 1.0) we reduce the training set size for that dataset, marked in Table 1. We focus on small datasets with up to 2 000 samples in total, because feature engineering is most important and significant for smaller datasets.
We describe the collection and preprocessing of datasets in detail in Appendix G.I.
http:///gghtcccccoooo//////////a inneeeeenn...rreeeeeenneeeeeeee py htttpp///ggtaab..oo/aall///////e
https: http:///w.....r..........oo.oooo atttfffat

---

Table I: ROC AUC OVO results using TabPFN. = indicates the standard deviation across splits. [R] indicates datasets where reduced data was used because TabPEN had 100% accuracy by default, see Appendix G.1.
TabPFN
No Feat. Eng.
CAAFE (GPT-3.5)
CAAFE (GPT-4)
airlines
0.6211 1.04
0.619 1.04
0,6203 1.04
halance-scale [R]
0.8444 1.29
0.844 2.31
0.882 1.26
breast-w [R]
0.9783 1.02
0.9809
0.9809
cmc
0.7375 1.02
0.7383 #.02
0.7393 1.02
credit-g
0.7824 +.03
0,7824 :0.
0.7832 2.03
diabetes
0,8427 +.03
0,8434 :00
0,8425 #.03
eucalyptus
0.9319 1:01
0,9317
0,9319
jungle_chess..
0.9334 1:01
0,9361
0.9453
pel
0,9035 =:01
0,9087 -20
0.9093 =.01
tic-tac-toe [R]
0.6989 +.08
0,6989 -00
0,9536 #.06
(Kaggle)
health-insurance
0.5745 1.02
0.5745 1.02
0.5748 1.02
Kaggle) pharyngitis
0.6976 1.03
0.6976
0.7078 1.04
Kaggle) kidney-stone
0.7883 1.04
0.7873 1.04
0.7903 1.04
(K aggle)
spaceship-tiantt titanic
0.838 #.02
0,8383 0202
0.8405 #.02
Evaluation Protocol For each dataset, evaluate 5 repetitions, each with different random seed and train- and test split to reduce the variance stemming from these splits (Bouthillier et al., 2021). We split into 50% train and 50% test samples and all methods used the same splits.
5
Results
In this section we showcase the results of our method in three different ways. First, we show that CAAFE can improve the performance of state-of-the-art classifier. Next, we show how CAAFE interacts with traditional automatic feature engineering methods and conclude with examples of the features that CAAFE creates.
Table 2: Mean ROC AUC and average rank (ROC AUC) per downstream classification method and feature extension method, Best AutoFE method per base classifer is shown in bold. The features generated by CAAFE are chosen with TabPFN as classifier. Rank are calculated across all classifiers and feature engineering methods. FETCH was too computationally expensive compute for all base classifiers in the rebuttal. Each seed and dataset takes to 24 hours and has to be evaluated for each base classifer independently. Thus, we use features computed for logistic regression for all other classifiers.
Baselines
CAAFE
No FE
DFS
AutoFeat
FETCH
OpenFT:
GPT:3.5
GPT4
Log, Reg.
0.764
0.754
0.757
0.763
0,769
Mean Rank
27.4
23.6
26.2
25.2
25
24.8
24.3
Random
Mean
0.783
0.783
0,785
0.785
0.803
Mean Rank
23.4
22.1
21.8
23.5
22.3
23.1
19.9
ASKL2
Mean
0.807
0.801
0.808
0.807
0.806
0.815
0.818
Mean Rank
12.2
12.9
12.6
13.4
13.5
10.9
11.6
Autogluon
Mean
0.796
0.799
0.797
0.787
0.798
0.803
0,812
Mean
TabPEN
Mean
0.798
0.791
0.796
0.796
0.798
0.806
0.822
Performance of CAAFE CAAFE can improve our strongest classifier, TabPFN, substantially. If it used with GPT-4, we improve average ROC AUC performance from 0.798 to 0.822, as shown in Table 2, and enhance the performance for 11/14 datasets. On the evaluated datascts, this improvement
7

---

Table 3: Examples of common strategies employed by CAAFE for feature extension. The full code and comments are automatically generated based on the user-por user-provided dataset descriptions. CAAFE combines features, creates ordinal versions of numerical features through binning, performs string transformations, removes superfluous features, and even recovers from errors when generating invalid code.
Description
Generated code
Combination
Example from the Kaggle Kidney Stone dataser.
Usefulness: Fever and rhinorrhea are two of the most common symptoms of respiratory infections.. including GAS pharyngitis. This Teature caplures their co-occurrence.
Input samples: "temperature [38.0, 39.0, 39.51, "thinorrhea ': [0.0, 0.0 0.0]
df [el''fver_adde ann_ondddorrreaaa '] ((dfl temperature '] >= 38.0) (dfl' rhinorrhea 0)) aittpe
Binning
Example from the Kaggle Spaceship Titanic dataset.
Feature: AgeGroup cattggorizes passengers into age groups) Uxefulness: Different age groups might have different likelihoods of being transported.
# Input samples: Age : [30.0, 0.0, 37.0]
bins [0, 12, 18, 35, labels [" Child", 'Teen" 'YoungAdult', 'Adult', 'Senior"1 dfl 'AgeGroup I = cut 'Aee bins=bins. labels=labols)
100]
arl AgeGGrop df[ AggGrroup astype "aalegory
String transformation
Example from the Kaggle Spaceship TTitanic dataset.
* Feature: Deek
The deek information can help identify patterns in the location eabins associated with transported passengers. * Input samples: "Cabin': ('F/356/8', "G/86/P" 'C/37/P"] drl Deck'I dfl "Cain I.apply flambda x101 if isinstance str) else 'Unknown')
Uxefulness:
# Feature: CabinSide
Usefulness:
The side of the cahin can help identify patterns in the location cabins associated with transported passengers.
# Input samples: "Cabin :: ('F/356/S' "G/86/P", "C/37/P" df[ CabinSide df[ Cabin a. (lambda ("1 if isinstance str) else 'Unknown')
Removing features
Drap original columns
Example from the Balance Scale
Explanation: original columns 'left -weight 'left "distancc', right-weight ', and right-distance can be dropped as the newly created columns fefi_mmmmett.. "right_moment" and differenceee capture the relevant information for predicting the class df. doopmmol (ooteoosssssllttteeeeee right -distance ], inplace=True)
datuset.
-lsttddittace', 'right-weight",
No more additional columns needed
Error Recovery
Feature: dif_eeifereence
Example from the Kaggle Health In- surance dataset. The code execution failed bocause missing values not considered in the dataframe. In suggested
Usefulness: Age difference between upper lower age can be
useful in determining the likelihood of person applying for policy.
Input samples: 'Upper_Age": [3C3'', 'C2', CC3', 'Lower_Age"s Owned "Rented" "Rented
iteration that can handle missing valiee.
'Age_ differeecee drl Upeer_Agee extraett astype) int) df[' Lowerr_ extrract (Id+)") astype (ini)
Feedback: failed with error: cannot convert float NaN integer.
Feature: Age_ differreeeec
* Usefulness: Age difference between upper and lower age can be useful determining likelihoud person applying policy.
Input samples: 'Upper_Age ["C33*, 'C2', "C"", 'Lower_Age P' Owned ", "Rented" "Rented
"Cl'Age. dig__difference 'I arl Upeer__gee str. extraet (Id+))) astype float). fillna (0) df[ ALower_Ago I. extract( (Id+) astype float). fillna (0)

---

is similar (71%) to the average improvement achieved by using a random forest (AUC 0. 0.73)) instead of logistic regression (AUC 0.749). We can see that CAAFE even improves performance for all of the new datasets from Kaggle. If we use CAAFE with GPT-3.5 only, we can see that it performs clearly worse than with GPT-4, and only improves performance on 6/14 datasets.
There is great variability in the improvement size depending on whether (1) problem is amenable to feature engineering, i.e. there mapping of features that explains the data better and that can be expressed through simple code; and (2) the quality of the dataset description (e.g., the balance-scale dataset contains an accurate description of how the dataset was constructed) Per dataset performance can be found in Table 1. CAAFE takes 4:43 minutes to run on each dataset, 90% of the time is spent on the LLM's code generation and 10% on the evaluation of the generated features. In Appendix F we plot the performance, time and cost of CAAFE across feature engineering iterations, showing the tradeoff between these parameters. For the 14 datasets, splits and 10 CAAFE iterations, CAAFE generates 52 faulty features (7.4%) in the generation stage, from which it recovers (see Figure 3).
Incorporating Classical AutoFE Methods Classical AutoFE methods can readily be combined with our method, one simply runs CAAFE first and then lets a classical AutoFE method find further feature extensions, as we did in Table For less powerful downstream classifiers, namely Logistic Regression and Random Forests, we observe that applying AutoFE additionally to CAAFE improves performance further. The AutoMI. method TabPFN on the other hand not improved by applying the evaluated AutoFE methods. This discrepancy might stem from the larger hypothesis space (complexity) of TabPFN, can get all necessary information from the data directly. For all combinations of classifiers and additional AutoFE methods, we can see that CAAFE improves performance on average.
Conclusion
Our study presents a novel approach integrating domain knowledge into the AutoML process through Context-Aware Automated Feature Engineering (CAAFE). By leveraging the power of large language models, CAAFE automates feature engineering for tabular datasets, generating semantically meaningful features and explanations of their utility. Our evaluation demonstrates the effectiveness of this approach, which complements existing automated feature engineering and AutoML methods.
This work emphasizes the importance of context- -aware solutions in achieving robust outcomes. We expect that L.MSs will also be useful for automating other aspects of the data science pipeline, such as data collection, processing, model building, and deployment. As large language models continue to improve, it is expected that the effectiveness of CAAFE will also increase.
Dataset descriptions play a critical role in our method; however, in our study, they were derived solely from web-crawled text associated with public datasets. If users were to provide more accurate and detailed descriptions, the effectiveness of our approach could be significantly improved.
However, our current approach has some limitations. Handling datasets with large number of features can lead very large prompts, which can be challenging for LLMs to process effectively. The testing procedure for adding features is not based on statistical tests, and could be improved using techniques of previous feature engineering works. L.Ms,, at times, exhibit a phenomenon known as "hallucinations." where models produce inaccurate or invented information. Within CAAFE, this might result in the generation of features and associated explanations that appear significant and are logically presented, even though they may not be grounded in reality. Such behavior can be problematic, especially when individuals place trust in these systems for essential peccisionmmaking or research tasks. Finally, the usage of LLMs in automated data analysis comes with a set of societal and ethical challenges. Please see Section B for a discussion on the broader impact and ethical considerations.
Future research may explore prompt tuning, fine-tuning language models, and automatically incor- porating domain-knowledge into models in other ways. Also, there may lie greater value in the interaction of human users with such automated methods, also termed -uman-in--he-looe AutoML (Lee & Macke, 2020), where human and algorithm interact continuously. This would be particularly easy with a setup similar to CAAFE, as the input and output of the LLM are interpretable and easily modified by experts.

---

References
Anaconda. The state of data science 2020. Wehsite, 2020. URL htt.:httss:/ww..aacccc.t com/
Slate-of-data-scaee setttta--ceenne2222222 2020.
Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand, Edward Raff, Kanika Madan, Vikram Voleti, Samira Ebrahimi Kahou, Vincent Michalski, Tal Arbel, Chris Pal, Gael Varoquaux, and Pascal Vin- cent. Accounting for variance machine learning benchmarks. In A. Smola, A. Dimakis, and I. Stoica (eds.), Proceedings of Machine Learning and Systems, volume 3, pp. 747- 769, 2021. URL
mwsys. http:/ape___fifee http:/aapppp/////eee pdf.
cf
Leo Breiman. Random forests. Machine learning, 45:5-32, 2001.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neurul information processing systems, 13:1877-1901,202
Adam tack
Crockett.
Ai generated vector, April 2023.
code
new security at- URL https://dev.to. wtw.taaaaaeeeeeeeeeeeeeeeaeeaaee cactamemccloees/////
creates
ai ti--gnnerated-coo coweraated-ood--eeee ceeee-crraeeeeeee a- wwwwrsscuuuuteeeeee
Tijl De Bie, Luc De Raedt, José Hernández-Orie Orallo, Holger H Hoos. Padhraic Smyth, and Christo- pher KI Williams. Automating data science. Communications of the ACM, 65(3):76-87, 87, 2022.
Matthias Feurer, Jan N. van Rijn, Arlind Kadra, Pieter Gijsbers, Neeratyoy Mallik, Sahithya Ravi, Andreas Mueller, Joaquin Vanschoren, and Frank Hutter. Openml-python: an extensible python api for openml. arXiv, 1911.02490. URL https: /tttps://arztie org/ 1911/.111...11 1949. ....99..dd.. pdf.
Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. Tabllm: Few-shot classification of tabular data with large language models, 2023.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. CoRR, 0os/2103.038744 2021. URL httph:tt//// //arxiv. org/ http://11...0777.
Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. Tabpfn: A transformer that solves small tabular classification problems in second. arXiv preprint a8X18 2022.
Franziska Horn, Robert Pack, and Michael Rieger. The autofeat python library for automatic feature engineering and selection. CoRR, abs/1901.00 .07329, 2019. URL http::::/axxii.org/bbbbbe 07329.
F. Hutter, L. Kotthoff, and J. Vanschoren (eds.). Automated Machine Learning: Methods, Systems, Challenges. Springer, 2019. Available for free http://auttmll.rg/bbook.
James Max Kanter and Kalyan Veeramachaneni. Deep feature synthesis: Towards automating data science endeavors. In 20/5 IEEE international conference on data science and advanced analytics (DSAA), pp. 1-10. IEEE, 2015.
Udayan Khurana, Deepak Turaga, Horst Samulowitz, and Srinivasan Parthasrathy. Cognito: Auto- mated feature engineering for supervised learning. In 2016 IEEE 16th International Conference Data Mining Workshops (ICDMW), pp. 1304-1307, 2016. doi: 10. 10.1109/CDM..2016.000
Udayan Khurana, Horst Samulowitz, and Deepak Turaga. Feature engineering for predictive modeling using reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
Doris Jung-Lin Lee and Stephen Macke. A human-in-the-logg perspective automl: Milestones and the road ahead. IEEE Data Engineering Bulletin, 2020.
Avanika Narayan, Ines Chami, Laurel Orr, Simran Arora, and Christopher Ré. Can foundation models wrangle your data?, 2022.
10

---

Fatemeh Nargesian, Horst Samulowitz, Udayan Khurana, Elias B Khalil, and Deepak S Turaga. Learning feature engineering for classification. Ijcai, volume 17, pp. 2529-2535, 2017.
OpenAl. Gpt-4 technical report, 2023a.
OpenAI. openaa/oppemmii-cooooooo Examples and guides for using the openai api. https:///iihhu.. http://paaaa/ppeaaacccobobbbbbba 2023b. (Accessed on 04/20/2023).
OpenAl Community. GPT-3 can't syllables doesn't I "get" haiku. https://ccomiii openai wttpt/gptt/ttttecee 3-cant--cee conaatt-coontt-yllallll..eeeee or aor-doent--gen Beet-haiku/18777 2021. Ac- cessed on: 2023-03-21.
Vinay Uday Prabhu and Abeba Birhane. Large image datasets: pyrrhic win for computer vision?,
2020.
Inioluwa Deborah Raji, Emily M. Bender, Amandalynne Paullada, Emily Denton, and Alex Hanna. Ai and the everything the whole wide world benchmark, 2021.
Chris Rohlf. Ai code generation and cybersecurity, April 2023. URL httph::t////ttt cfr. org/
blog/ hetg/ai-cone code-
Alaina N. Talboy and Elizabeth Fuller. Challenging the appearance of machine intelligence: Cognitive bias in Ilms and best practices for adoption, 2023.
Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo Openml: Networked science in machine learning. SIGKDD Explorations, 15(2):49-60, 2013. doi: 10.1145/2641110.26411111 URL hwtttttppp/.o..// a.m. org/ 10. 1..0..14:00 1145/26411999 2641198.
Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luís Torgo Openml: networked science in machine learning. CoRR, abs/1401 0101 407.7722, 2014. URL httt:://axiiii warrg/bss/iee 1107. .7722.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/1 2017. URL
http:// arxiv. org. httg//ab///// 1706.000000 037622.
David Vos, Till Döhmen, and Sebastian Schelter. Towards parameter-efficient automation of data wrangling tasks with prefix-tuning. In NeurlPS 2022 First Table Representation Workshop, 2022.
URL https:t////// http///ppeeeeiieeeee/eeee net aet//orummidd-BkyY J52KH..
Hilde Weerts, Florian Pfisterer, Matthias Feurer, Katharina Eggensperger, Edward Bergman, Noor Awad, Joaquin Vanschoren, Mykola Pechenizkiy, Bernd Bischl, and Frank Hutter. Can fairness be automated? guidelines and opportunities for fairness-aware automl, 2023.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2/09. 016222,, 2021.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023.
Wes McKinney. Data Structures for Statistical Computing in Python. In Stéfan van der Walt and Jarrod Millman (eds.), Proceedings of the 9th Python Science Conference, pp. 56 61, 2010. doi: 10.25080/Majora- MaA.aaajjrraa2b1192220
Svante Wold, Kim Esbensen, and Paul Geladi. Principal component analysis. Chemometrics and intelligent laboratory systems, 2(1-3):37-52, 1987.
Jianyu Zhang, Jianye Hao, Françoise Fogelman-Soulié, and Zan Wang. Automatic feature engi- neering by deep reinforcement learning. In Proceedings of the 18th International Conference Autonomous Agents and MultiAgent Systems, pp. 2312-2314, 2019.
Daniel M Ziegler. Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei. Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909 08593, 2019.

---

A
Acknowledgements
GPT-4 (OpenAI, 2023a) was used in the following ways: to help us iterate LaTeX formatting and diagram plotting; for text summarization; and as a copyediting tool; for rephrasing proposals.
B Broader Impact Statement
B.1 Social Impact of Automation
The broader implications of our research may contribute to the automation of data science tasks, potentially displacing workers in the field. However, CAAFE crucially depends on the users inputs for feature generation and processing and provides an example of human-in-the-lopp AutoMI. The automation of routine tasks could free up data scientists to focus on higher-level problem-solving and decision-making activities. It is essential for stakeholders to be aware of these potential consequences, and to consider strategies for workforce education and adaptation to ensure a smooth transition as Al technologies continue to evolve.
B.2 Replication of Biases
Al algorithms have been observed to replicate and perpetuate biases observed in their training data distribution. CAAFE leverages GPT-4, which has been trained on web crawled data that contains existing social biases and generated features may be biased on these biases. An example study of such biases in web-crawled data was done by Prabhu Birhane (2020). When data that contains demographic information other data that can potentially be used discriminate against groups, advise not to use CAAFE or to proceed with great caution, double checking the generated features.
We would like to mention the best practice studies fairness for AutoML by Weerts et al. (2023) and on bias in the use of LLMS by Talboy Fuller (2023).
Weerts et al. (2023) give recommendations for what AutoMI. developers can do to improve fairness- conscious AutoML, and follow the three most emphasized closely. i) We clearly emphasize the limitations and biases of our method. i) We emphasize the 'Usefulness' comment that our prompt asks the LLM to write into our UI "help users identify sources of fairness-related harm", Weerts et al. (2023) write. ii) We execute code only after asking the user for explicit confirmation that it does not contain dangerous actions or biases; these kinds of design decisions are called "seam-fite full design" by (Weerts et al., 2023).
(Talboy & Fuller, 2023) give recommendations how best use LLMs. Their recommendation, which we as interface developers can more easily follow, is the one they emphasize the most. Namely, that "LLMM [...] should be used decision support tools, not final decision makers. We follow this closely by not allowing the LLM to execute code without the user's explicit consent.
The choice of benchmarks can also introduce dangerous biases in machine learning methods, as there are only a few that dominate the landscape (Raji et al., 2021). We address this problem by using datasets from two different sources (Kaggle and OpenML). In addition, we multiple datasets from each source.
How Can Biases Arise in CAAFE? Since are using trained model (the LLM) define the training setup (features used) of a downstream classifier trained on potentially biased data, we can have bias at three levels: i) at the level of the LLM, and ii) at the level of the generated features, ii) at the level of the downstream classifier.
The level of the downstream classifier is affected by the previous two levels, but is not directly affected by CAAFE, so we do not discuss it in detail. Here, as in any data science application, the must be careful to use unbiased data. The way set up CAAFE has some robustness against biases the first two steps, which are controlled by CAAFE:
i) We use GPT-4, where great care has been taken to mitigate biases - still it does contain biases in its generations. We add a statement that choosing a language model that trained to mitigate biases is crucial step for the user to tackle this problem.
12

---

ii) Feature engineering operations by the LLM are only retained if they lead to improvements in voss-valid,,ion, due the way our method is set up (we generate code and verify that improves performance at each step). This gives us slightly better defense against bias than using the LLM outputs directly, since biases that are present the LLM but not in the data are discarded.
However, since the biases of the LLM and the dataset are most likely not independent, there is still risk that the two will be aligned. To illustrate this risk and make users aware of potential problems, we include an illustrative Example (see end of this reply).
Simple Example We have built a fake dataset that has only one feature, a person's 's name, and as output want to predict whether the person doctor or nurse. The description of the dataset used in our challenge is
Doctor-ord -or-orre is dataset that aske to predict whether person is doctor or a nurse just based their name.
the prompt ia Columne in 'df' (true feature dtypes lieted here, categoricale encoded as int): (object): NaN-freq [o. [X] Samples ['Anna', , 'Frank', 'Laura', "Travis", Sophia']
intentionally
sample the
to
ie_dector (int): MaN-freq [0.0%] Samples [o, 1, 1, 0]
In cases this sample name commonly associated with male gender is associated with a doctor. Additionally, we only used female names ending in "a".
We that GPT-4 tends suggest an attribute that the typically gender-associated ending of the name to classify, i.c. it generates the following output
Feature: end_vith._
Usefulness: Names ending with night
lensles. Aseuning there might be gender bias in the doctor or nurse profeasion, this could provide some useful information.
Input samples: [>Anns', 'Jack', "Frank"] ']
df ''d__with_a'] df ('amne a...ttre.ddiiii asttt
Listing 1: CAAFE output
We can see that our prompt actually makes it very easy to detect the bias the LLM this case, since the "Usefulness' comment the LLM is asked to provide, least in this non-cherry-pice -iccked example, just gives away its bias.
B.3 AI Model Interpretability
As the adoption of advanced Al methods grows, it becomes increasingly important to comprehend and interpret their results. Our approach aims enhance interpretability by providing clear explanations of model outputs and generating simple code, thus making the automated feature engineering process more transparent.
B.4 Risk of increasing Al capabilities
We do not believe this research affects the general capabilities of LLMSs but rather demonstrates their application. As such we estimate our work does not contribute to the risk of increasing AI capabilities.
C Reproducibility
Code release In an effort to ensure reproducibility, we release code to reproduce our experiments whtttt://// :// thub. com/ automl/CAAAE We release minimal demo simple demo.
Availability of datasets All datasets used in our experiments are freely available at OpenML. org (Van- schoren al., 2014) or at kaggle, com, with downloading procedures included in the submission.
13

---

D Full LLM Prompt
Figure 5 shows the full prompt for one examplary dataset. The generated prompts are in repository:
https:/ //gs thub. com/ /ttaa//////AAA///eeeeeaaaaaaa/aa httpa///aaaaaaeedeeeeeeeeo generated_code
The dataframe 'df' loaded and memory. Columns are also named attributes. Description of the dataset in "dr" (column diypes might inaccurate): Te=TicTTac-To Endgame database
This
encodes toe games. where "x" is assumed to have played first. The larget concept is "win for x" (i.e.. true when
complete set possible board configurations
-tae
one possible ways to create a "three -in-a-row")
Columns 'df' (Irue feature diypes listed here. categoricals encoded as int):
top- top-middle-square (int32): NaN-freq [0.0%], Samples [0, 0, 1, 1, 2,
lop-left -qquare (int32); NaN-freq [0.0%], Sumples [2. 2, 2.
2.
2,
0,
1,
0, 2.
sup-right-square (inl32); NaN-frey [0.0%]. Samples
0,
2.
2.
middle-left -qquare (int32): NaN-freq [0.0%]. I. Samples [1, 0, 2.
1, 2,
2]
middle-miidee-square (int32): NaN-freq [0.0%]. Samples [0, 2. 1,
middlee-right-square (int32): NaN-freq [0.0%]. Samples [1. 1.
bottom-leit-squuree (in132): : NaN-freq 10.0%). Samples 12. 1. 1,
1.
battom-middle-squuree (int32): NaN-freq [0.0%], Samples [2, 0,
2,
bottom-right-squar (int32): NaN-freq [0.0%], Samples [2, 2, 0,
0, 2. 2,
(category): NaN-freq [0.0%]. Samples [1.0,
1.0, 1.0, 1.0, 0.0. 1.0, 0.0, 0.0, 0.0]
This code written
expert datascientist working to improve predictions. is snippel of cade that adds new columns lo the dataset.
Number of samples (rows) training dataset: 71
This code generates additional columes that are useful algorithm (such as XGBeost) predicting "Class"
downstream classification
Additional columns add new semantic information that is they use real world knowledge on the datasel. They e.g. be feature combinations. transformations. aggregations where the new column is function of the existing columns.
The scale columns and offset does
all used columns exist. Follow the above deseription of columns closely and consider the datatypes and meanings of classes. may redundant
Make
This code
draps columns, the downstream classifier (Feature selection). Dropping columas may help the chance overfitting lower, especially if the dataset is small.
hurt predictive performance
The classifier will be trained
the dataset with the generated calumns and evaluated
holdout set. The evaluation metric is accuracy. The best performing cade will be selected
Added columns can be used in other codeblocks. dropped columas are not available anymore.
Code formatting for each added column:
"pythen
(Feature and description)
Usefulness: (Description why this adds useful real world knowledge to classify "Class" according dataset deseription and attributes.)
Input samples: ( (Three samples of the columns used in the following code, e.g. 'top-left- square [2. 2, 21,
square [0. 0. 1]. ...).
add column for Some pandas code using top-teft-squat "top-middle mmniddle-squree each row d()
. 'end
Code formatting for dropping columns: "'python
XX is dropped dr. croo(coouums=| 'X"I. implace=True) "eend
Explanation why
Each codeblock generates exactly one useful columa and can drop unused columns (Feature selection).
Each cadeblack ends with eeend and starts with """'phhho"" Codeblock:
Figure 5: Full LLM Prompt for the CMC dataset. The generated code will be the reply this prompt.
14

---

E
Additional Results
E.1 Semantic Blinding
Semantic information, i.e. the context of the dataset and its columns, is crucial and can only be captured through laborious human work or our novel approach using LLMs this is the core of our approach. To further verify and quantify this claim, we perform experiment where the context of the dataset is left out (i.e. feature names and dataset description are not given to the LLM). We find strong drop in performance from average AUROC of 0.822 to 0.8 over all datasets for GPT-4.
Faating"".FFo For "Semantic Blinding" feature names and the dataset description is concealed to CAAFE. Mean ROC AUC and average rank (ROC AUC) per downstream classification method and feature extension method is shown. Best performing AutoFE method per classifier shown in bild. The features generated by CAAFE are chosen with TabPFN as classifier. Ranks are calculated across all classifiers and feature engineering methods.
Table 4: CAAFE with semantic and without "Semantic Blinding".
GPF:3.5 Semantic Blinding
No FE
Default
Semantic Blinding
Default
Log, Reg.
Mean
0,749
0.754
0.763
0.749
0.769
Mcan Rank
19.6
19.1
17.8
19.3
17.4
Random Forest
Mean
0,782
0,789
0.79
0.783
0.803
Rank
17.2
16.5
16.8
16.5
ASKL2
Mean
0.807
0,812
0.815
0.809
0.818
Autogluon
Mean
0.796
0.803
0.803
0.801
0.812
Mean Rank
12.9
12.2
11.6
12.7
10.3
TμPPEN
0.798
0.807
0.806
0.822
Meun Rank
10.5
9.33
9.71
9.33
7.59
E.2
Per Dataset Results
Table 5: ROC AUC OVO results per dataset and downstream classification method. CAAFE optimized for strong performance TabPFN.
AutoFE-Bast Base
CAAFE
Bass CAAFE
AutoFFeat
GPT:3.5
GPT
airlines
0.6211 04
0.6076 8.04
0.595 + 044
0.619 1:04
0.6203 : 04
0.602 + 04
0.5966 +
belance-scale
0.8444
0.8438
0. 0.8428
0.844 1.31
0.882
0.8812 1.27
0.8773
breast-w
0.9783 1.02
t.03 0.9713
0.9783 0.02
0,9809 :: 02
0.9809 0:02
0.9713 8:03
0,9809 :: 02
credit-g
0.7824 .0
0.7819 1.03
0.7824 1
0.7824 1:
0.7832 1.03
0.784 1.00
0.7824 1
diabetes
0,8427 000
0,8414 4.05
0.8417
+ 03
0.8434 +
0.8425 0:03
0.8432 0:03
0.8382 + 03
eucalyptus
0.9319
0.9321
0.9319
+
0.9317
0.9319
0.9323
0.9319 +
jungle_hher chess..
0.9334 1.01
0.9197 1.
0.9284
+
0.9361
+ 01
0.9453 01
0.9535 1: 01
0.94 11:
(K Kaggle) bealth-insurance
0,5805
0,5745
0.5748
0.5777
0.5782
[K Kaggle) pharyngitis
0.6976 0:03
0.6976 4:03
0.6976 4:03
0.6976 + 03
0,7078 0:04
0.7073 4: 04
0.6976 +.
(Kaglle) kidney
(agglee spaceship-titanic
0,838 02
0,8486 #:02
0.8443 1
0.8383 +
0.8405 - 02
0,853 + 02
0.8486 1
pel
0.9035 4.01
0.9046 4.01
D. 0.9035 01
0.9057 + 02
0.9093 00
0.908 4:00
D. 0.9035 + 01
tic-tac-tne
0.6989
0.6989
0.6291
0.6989 +
0.9536
0.9536 2:
0.938
E.3 Generated Prompts and Code
You can find the generated prompts the respective L.LM generated code our repository: attps://githur commcaafautooateedee wawteaaeeeeeeeeeeeeeeeeea
eatures wtw.uurreee//e/eeeeeeeeeeeeeeeee htt.///treeeeeaiieeeeeeeeeeeeeee aalaannnaattt datta//ggnnrtaed_ooee
15

---

Compute
5
0.8225
Mean ROC AUC
0.8200
Time in minutes
costin in
0.8175
Token cost
0.8150
0.8125
0.2
0.8100
0.8075
0.0
1
3
8
10
Number of iterations
Figure 6: Mean ROC AUC OVO, inference cost for GPT and time spent with an increasing number of feature generation runs.
Figure F illustrates the increasing performance but also cost and time spent for more feature engineer- ing iterations. Prediction for LLMs done per token and so the generation of code takes dominates the 4:43 minutes evaluation time of CAAFE average per dataset. For GPT-3.5 this time is reduce to about 1/4, Also for GPT-3.5 the cost reduced to 1/10 as of the writing of this paper. For evaluation of TabPFN 2.80GHz CPU cores.
Nvidia RTX 2080 Ti well as 8 Intel(R) Xeon(R) Gold 6242 CPU
G Datasets
# Features
# Samples
Classes
OpenML ID/ Kaggle Name
Name
balance-scale
125
3
11
breast-w
69
2
15
eme
1473
3
23
credit-g
20
1000
2
31
diabetes
768
37
tic-tac-toe
95
2
50
cucalyptus
19
736
5
188
pel
21
1109
2
1068
airlines
2000
2
1169
2000
3
41027
pharyngitis
19
512
2
pharyngttis
health-insurance
13
2000
2
spaceship-tiaane titanic
13
2000
2
spaceship titanic
kidney-stone
414
2
playground-sert series-33e12
Table 6: Test datasets used for the evaluation. See Section 4 for description of the datasets used.
G.1 Dataset Collection and Preprocessing
OpenML datasets We use small datasets from OpenML (Vanschoren al., 2013; Feurer al.) that have descriptive feature names (i.e. we do not include any datasets with numbered feature names). Datasets on OpenML contain task description that we provide as user context to our method and that we clean from redundant information for feature engineering, such as author names or release history. While some descriptions are very informative, other descriptions contain much less information. We remove datasets with more than 20 features, since the prompt length rises linearly with the number
16

---

balanced, The attributes and the right distance, The correct way to find the class is the greater of (left- right-seight). If they equal, it of features and exceeds the permissible 8, 192 tokens that standard GPT-4 can accept. We show all datasets we used Table in Appendix G. When datasets are perfectly solvable with TabPFN alone (i.e. reaches ROC AUC of 1.0) we reduce the training set size for that dataset to 10% or 20% of the original dataset size. This the case for the datasets "balance-scad "tic-tac-toe" (10%). We focus on small datasets with up to 2 000 samples in total, because feature engineering most important and significant for smaller datasets.
(20%), "breast-w" (10%) and
Kaggle datasets We additionally evaluate CAAFE on 4 datasets from Kaggle that were released after the knowledge cutoff our LLM Model. These datasets contain string features as well, String features allow for more complex feature transformations, such separating Names into First and Last Names, which allows grouping families. We drop rows that contain missing values for our evaluations. Details of these datasets can also be found in Table 6 in Appendix G.
G.2 Dataset Descriptions
The dataset descriptions used were crawled from the respective datasource. For OpenML prompts uninformative information such as the source or reference papers were removed. Figures 20 show the parsed dataset descriptions used for each dataset.
Balancce Veight Distance Database**
This data set was generated to nodel paychological experimental results. Each example is having the balance scale tip the left veight, the left distance, the right weight,
clasaified
right, tip left, or
distance left refghteight) and Fright-distane is balanced.
Attribute description
The attributes are the left veight, the left distance, the right seight, the right distance.
Figure 7: Dataset description for balance-scale.
**Breast Cancer Visconsin (Uriginal) Data Set.** Features are computed from digitized image needle sepirate (FNA) characteristica of the cell nuclet present in the image. The target feature records the prognosis (nalignent or benign).
mase. They describe
Figure Dataset description for breast-w.
17

---

4. Relevant Information:
1987 National Prevalence Survey. The samples narried who vere either not pregnant or do not know if they were at the time of interviev. The problen (no use, long-tera nethods, or short-tera nethods) of xonan based cbaracteristics.
This datasst aubset
Contraceptive
predict
contraceptive nethod
demographic
T. Attribute Information:
1. Vife's age
(aumerical)
education 3. Husband's education
(categorical)
1-1ow. , 3, 4-high
(categorical)
1-1ov, 2, 3, 4-high
(numerical)
5. Vita's religion
(binary) (binary)
0=Xon- Ialam, 1=lalan
Vife's now working? Husband'e occupation
O-Yes, 1-No
(estegorical) (categorical)
2. 3, 4
Standard -living index
1-1oz, 2, 3, 4-high
Nedia exposure
(binary)
O-Good, 1-Not good 1-Xo-use
10. Contraceptive method used
(class attribute)
2=Long tern 3=Short -ter
Figure 9; Dataset description for emc.
This dataset classifies people described by set of attributes as good or bad credit risks.
This dataset comes with cost natrix: cre
Good Bad (predicted)
(actual)
Bad 5 cre
It is worse to class custoner as good when they are bad (5), than 1s to clase they good
description
1. Status of ezieting checking account, in Deuteche Mark. nontha
3. Credit history (credits taken, paid back duly, delaye, critical accounts) (car, television....
Purpose
6. Credit anount
Statua savinga account/bonda, Present employment, in number of years. Inetallment rate in percentage of dieporable incone (narried, single,.... 10. Other debtors / guarantore residence 12. Proparty (e.g. real estate)
Deutache Marx.
years
Age years
14. Other installuent plane (banks, stores)
15. Heusing (rent, ozn....)
people boing this 19. Telephone (yas,no)
Bunber existing
17. Jab
provide
20. Foreign vorker (yea no)
Figure 10: Dataset description for credit-g.
18

---

4. Relevant Information:
constraints sere placed larger database. In particular, all patients here are females at least years old of Pima Indien heritage. ADAP is an adaptive learning routine that
selection
instances
and executer digital analoge perceptron like devices. It is a unique algorithn; see the paper for details.
7. For Each Attribute: (all numeric-valued)
pregnant
2. Plasma glucose concentration a 2 hours in an oral glucose tolerance test Diastolic blood preseure (an lig)
Triceps skin fold thickness (mn)
2-Hour serun inaulin (mu u/ml)
(veight kg/(height a) 7. Diabetes pedigree function
Body
Age (years)
9. Glass variable (0 1)
Relabeled values attribute 'class'
From:
To: tegted_ bested_
Figure 11: Dataset description for diabetes.
*+Tic- Endgane This database encodes the complete set of possible board configurations at the and of tic -tac ganes, where "x" is aseuned have played first. The target concept is for (i.e., when "x" has of possible ways to create "three-in row").
Figure 12: Dataset description for tic-tac-toe.
19

---

B*acalyppus Soil Conservation The objective in seasonally dry hill country. Determination is round by measurement of height, by height, survival,
daternine which saedlots species bast for soll conservation
contributing factors.
It is inportant to note that eucalypt trial methods changed over tine; earlier trials included mostly 15 30cm tall seedling have included nostly three replications of eight trees groun. This change may contribute leas significant resulta.
in peat plots and the later trials
Experimental recording procedures
require noting include: instances with no data recorded due to experimental recording procedures require that the absence of a species fron one replicate at site was value, but if absent fron site the species was excluded from the site's analyses.
treated
replicates
aissing data and utility especially for the data recorded at the Morea Station; this species
vigour,
resistance,
indicate of data.
areas
Attribute Information
enunerated
2. Rep site rep integer
Locality site locality in the North Island enumerated Map_Ref location the North Island enumerated
5. Latitude latitude approxination enunerated approxination integer 7. Rainfall rainfell (mn pa) integer
Altitude
Frosts
(deg- z) integer
Year yyear of planting integer
10. Sp species code anumerated
PMCno
number
12. рвн best dianeter base height (ca) real
height
14. Surr survival integer
Vig vigour
16. Ins_rea insect reaistance real
17. Stem_Fn sten forn real
Crosn_Fn
19. Brach_Fn branch form real
20. Utility utility rating enumerated
papera
Bulluch (1992) Bacalyptus Species
Seasonally Dry Hill Country Twalfth Year Assessment New Zealand Journal of Forestry Science (1991)
21(1):
Kirsten Thonson and Robert J. NoQueen (1996) Eachine Learning Applied to Fourteen Agricultural https: :/w:://wwwww cs. waikato.ac ac. az/ul////ppp htt.a///bblccctiinsseeeee.e////e 199///99/////// publication:
University Waikato -eeuueen-eeuueeeeeeee -96. + the original
Report
Figure 13: Dataset description for eucalyptus.
Binarized of the original data (see version 1). The multi-class feature is converted to a tuo-class nominal target feature by re-labeling the najority positive (1p²) others negative (*x*), Originally converted by Quan Sun.
Figure 14: Dataset description for wine.
20

---

prediction
blank lines One of the NASA Metrica Data Progran defect data sets. Data from flight softuare for earth orbiting satellite. Dats cones fron McCabe and Halstead features extractore of to objectively
code. These features defined in the 70s in characteriza code features that are asseciated with nofteare quality.
Attribute Information
lac
nuneric McCabe 's line of code
2. v(g)
nunaric X NcCabe "cyclonatic complexity"
(g)
NcCabe "essential complexity"
4. (g
nuneric X McCabe *dasign complexity"
nuneric Halstead total operators operands
numeric X Halstead "volume"
nuneric X Halsteed "progran length"
Halstead "difficulty"
9.
nuneric X Halstead "Intelligence"
Halstead "effort"
11. b
numeric Halstead
12.
numeric Halstead's 's tine estimator
13. 10Code
numeric Halstead's line count numeric Halstead's count of lines of comments Halstead's
14. 10Comment
10Blank
16. 10CodeArdComment
numeric
uniq_0p
unique operstors numeric unique operands numeric total operators numeric total operands numeric X of the flor graph graph
18. uniq_Upmd
19. total_0p
20. total_Opad
21. branchGount
branchtount
23. defects
{Talse, true) module hae/hae not one or more reported defects
Relevant papers
Song Zhongbin Data Quality: Some Comnente on the BASA Software Defect Datasete, IESE Transactions Software Engineering.
Shepperd,
(2013)
Tim Menzies and Justin S. Di Stefano (2004) How Good is Your Blind Spot Sampling Policy 2004 IEEE Conference High Assurance Softsare Eagineering.
I. Menzies and J. DiStefano and Δ. Orrego and R. Chapman (2004) Aszessing Predictors of Software Nodels, Chicago
Workshop
Figure 15: Dataset description for pel.
Airlines Dataset Inspired in the regression dataset from Elena Ikonenovska. The task is predict given flight dalayad, given scheduled departure.
Figure 16: Dataset description for airlines.
21

---

Description
This dataset of collection datasets based the gane "Jungle Chess" (a.k.a. Dou Shou qi). For description of the rules, please refer to the paper (link attached). The paper also contains description of various constructed features. As the tablebases are diajoint set of saveral tablehases based on which (two) piaces board, have uploaded
that explicit different
content:
Bet Rat
Ret ve Panther
Rat Lien
Rat va. Elephant
Panther ve. Lion
Fanther ve. Elephant
Tiger va. Lion
Lien Lion
Lien vs. Elepbant
Elephant Elephant
Complete (Combination of the above)
Complete (Combination and strength information). This dataset contains a similar classification problen as e.g., the King and Rook ve. King problem and is suitable for claesification tasks.
above, containing
pieces just rank,
(Note that this dataset is one of the above mentioned datasets). Additionally, note that subproblens very aimilar. Having tablebases arguably gives a lot of information about the outcome of the same position
given peaitions fran
n. van Rijn Arliv link: https://arxiv. http:://araiv.orr/aah/1800 wwg/.b////0..00700333
Via, Endgame Analysia
Q1.
37:2,
120--124, 2014.
Figure 17: Dataset description for jungle_chess_@ppee cccceeeepesaa_eeen_eeeeeeee www_eangaaee_oopppeooo
objective, data in the Tabular Data format. policy recommended person chooses to fill form to apply, it is considered Poaitive outcone Classified as lead). All other conditions are considered Zero outcones.
Binary Claesification Problem
a person they on insurance vebsåte,
Figure 18; Dataset description for Kaggle health wnw.rranccceeeed-dediiiiiooooooo
Group A straptoceccua (GAS) infection ia major cause of padiatric pharyngitia, and infection with organien requires appropriate antimicrobial therapy.
whether physicians rely en signs symptons pediatric patiente with pharyagitis who should undergo rapid antigen detection testing (RADT) for GAS
controversy
Dur objective was to avaluate the validity of signs and symptoma in the selective teating children
pharyngitie.
learning analyze syпptons and signs. Can ve predict RADT positive?
diagnosis can
Figure Dataset description for Kaggle_pharyngitt.
22

---

Datazet Description
In this competition your task is to predict whether a passenger was transported to an alternate dimension during the Spaceship Titanic's collision with the spacetine anomaly. To help you make these predictiona, you 're given a aet of personal recorda ship's damaged computer systen.
File and Data Field Descriptions train.cev Personsl records for about woo-thirds ("8700) of the passengers, be used as training data.
Passengerid unique Id for passenger. Each Id the SEEE-PP 8868 indicates a group the paasenger is travelling with and pp is their nunber within the family nembere,
group. People HomePlanet The planet the passenger departed from, typically their planet of permanent residence.
group
always.
Cryošleep Indicates whether the passenger elected be put into suspended aninstion for the duration of the voyage. Passengers in cryosleep are confined to their cabins
Cabin - The cabin number where the passenger is staying. Takes the form deck/aun/side, can either Destination The planet the passenger will be debarking to.
or
Starbeard.
Age The age of the passenger.
VIP Whether the passenger has paid for special VIP RoonService, FoodCourt, ShoppingMall Spa, VRBeck Anount the paasengar has billed at Spaceship Titanic's many luxury anenities. Bane The first and last names of the passenger. Transported
during the voyage.
passenger transported target, the column are trying to predict.
dimension.
Figure 20: Dataset description for beegll_spaceshhp-titame
23


Reviews:
[{"review_id": "dBd4RHbRc7", "reviewer": "Reviewer_QFgW", "summary": "The paper proposed Context-Aware Automated Feature Engineering (CAAFE) approach to integrate new features learned from dataset descriptions using large language models into AutoML process for tabular datasets. The proposed approach was evaluated using 14 datasets.", "questions": "Highly recommend the author to run the data experiments using the existing AutoML systems (listed in weakness part, they are all open source) without LLM featurizations from dataset description as baselines.", "limitations": "Yes. The author addressed the limitations.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written with clear research motivation, experiment design and results illustration. It's an interesting attempt to try to integrate large language models into AutoML process. \nCode scripts are provided.", "weaknesses": "1. Overall the use case of adding features from data description is trivial, even using large language models or human-in-the-loop concept. The proposed feature engineering process through prompt generation from the data description or any semantic information looks more of heuristic rules in featurization, which could be performed without large language models. LLM plays the role of automated code generation, instead of machine learning feature engineering approaches (e.g. PCA) incorporated in most AutoML systems. \n2. The paper fails to catch up with the latest trends of AutoML systems. Most of existing AutoML systems will perform automated feature engineering, e.g. H2O AutoML, TPOT, AutoGluon, AutoSklearn, etc. Missing those AutoML baselines significantly eliminate the value of the proposed approach in modeling effectiveness. \n3. In Table 1, the increased CAAFE performance is relatively trivial for most datasets, except the tic-tac-toe, which significantly impact the mean ROC AUC.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the time and effort taken by Reviewer QFgW to evaluate our paper. We've carefully considered each point raised and aim to address them comprehensively below. We would also appreciate you reading through the general reviewer feedback, which outlines the changes we made and might address open questions that go beyond what you have asked for.\nFor this year's NeurIPS revisions, it is not possible to upload a modified paper, but only upload a one-page rebuttal with additional figures. So we posted changes that we will make in the final paper as part of our replies.\n\n### Weaknesses\n\n1. *\"Overall the use case of adding features from data description is trivial, even using large language models or human-in-the-loop concept. The proposed feature engineering process through prompt generation from the data description or any semantic information looks more of heuristic rules in featurization, which could be performed without large language models. LLM plays the role of automated code generation, instead of machine learning feature engineering approaches (e.g. PCA) incorporated in most AutoML systems.\"*\n\nWe believe that there is strong evidence that the generated features are not trivial and significantly benefit from the LLMs knowledge.\n\nBaseline approaches such as DFS, AutoFeat, FETCH and OpenFE perform worse than our approach, and we do not believe that features these approaches propose are trivial. Also, CAAFE improves even the performance of AutoML methods, such as Autogluon and Autosklearn, which already have built-in feature engineering. Please, see Table 2 in the rebuttal PDF for details. TLDR: CAAFE (GPT-4) + TabPFN is the strongest among all methods. Using TabPFN as a classifier, a critical difference diagram shows the statistically significant performance advantage of CAAFE to all baselines.\n\nSemantic information, i.e. the context of the dataset and its columns, is crucial and can only be captured through laborious human work or our novel approach of using LLMs - this is the core of our approach.\nTo further verify and quantify this claim, we perform an experiment where the context of the dataset is left out (i.e. feature names and dataset description are not given to the LLM). We find a strong drop in performance from an average AUROC of 0.822 to 0.8 over all datasets for GPT-4. Please, see Table 1 in the rebuttal PDF for details.\n\nWhy does semantic information help non-trivially?\n\nIt reduces computational complexity by considering useful features. We saw CAAFE generates sophisticated features depending on up to 9 base features.\nA featurization is only validated on the training data, but performance is measured on separate test data. Overfitting can occur in the same way as it does in training a model - a featurization can be viewed as just another step in the model construction. When the number of provided samples is small or the number of tested featurizations is large, the risk of overfitting becomes larger. By considering semantically meaningful features CAAFE generates fewer but more valuable features, that thus are more likely to generalize to the test set. \n\n2. You state that our work *\"fails to catch up with the latest trends of AutoML systems\"* since *\"most of existing AutoML systems will perform automated feature engineering\".*\n\nWe are very closely following the space of AutoML and are definitely aware of AutoFE being part of many AutoML approaches. The premise of our work is that classical AutoML and AutoFE cannot capture and use contextual information, which our work seeks to do. Our original manuscript contains TabPFN which is one such AutoML method, that implicitly engineers features. For this rebuttal, we evaluated further AutoML methods, namely Autosklearn and Autogluon, and use them together with CAAFE. Here we see that while baseline feature engineering methods do not improve AutoML methods, as the improvements made by them are already captured inside, CAAFE can improve even state-of-the-art AutoML methods. We explain this because the improvements gained by CAAFE come from semantic information (as we show above), which is not captured by current AutoML methods.\n\nPlease, see Table 2 in the rebuttal PDF for details. There you can see that CAAFE can improve upon AutoML methods (TabPFN, Autosklearn and Autogluon) in contrast to traditional AutoFE methods.\n\n\n3. *\"In Table 1, the increased CAAFE performance is relatively trivial for most datasets\"*\n\nThe improvement of CAAFE is similar to the improvement achieved by using a random forest instead of logistic regression on our datasets, which is a drastic difference. While mean AUROC is significantly affected by strong performance on tic-tac-toe, the number of wins and the ranks are not affected by outliers. We still see that CAAFE improves TabPFN on 11 out of 14 datasets, more wins than random forest has over logistic regression. It looks similar in terms of rank improvement: [Log. Reg. -> Rand. Forest: Mean rank 5.11 -> 4.44] and [TabPFN -> TabPFN + CAAFE: Mean rank 4.39 -> 3.06].\n\n### Questions\n\n1. *\"Highly recommend the author to run the data experiments using the existing AutoML systems (listed in weakness part, they are all open source) without LLM featurizations from dataset description as baselines.\"*\n\nThis question has been answered in part 1 of our \"Weaknesses\" replies.\n\n\nWe hope we could address your concerns and are open to additional insights or suggestions, ensuring our research stands robust in the domain.\n"}, {"review_id": "uBNAsDkSLH", "reviewer": "Reviewer_Y55o", "summary": "This paper presents a novel approach to automated feature engineering utilizing large language models (LLMs). Authors propose to use LLMs to generate code for feature generation. In the proposed method CAAFE, LLM is given a prompt with dataset description and a task of writing code for feature generation, the new feature is accepted if the validation score improves after training an ML model on a dataset containing the new feature. This procedure is repeated.\n\nThe paper demonstrates the effectiveness of the approach by testing on 10 public datasets from OpenML and 4 datasets from Kaggle (less likely to leak into the LLM training set) and slightly improving the overall classification accuracy of the strong tabular classification model TabPFN", "questions": "- What do the bold entries mean in the table? The standard deviation values are pretty large, but results are bolded. Were stat tests used?\n- Does performance saturate after 10 iterations? What is the reason for stopping there?\n- Do you update the prompt with the generated feature descriptions for new iterations (I think info is missing in the paper, but this seems important)?\n- Could you report some numerical quantification of features generated by CAAFE, like feature importance ranking?\n- Minor issues:\n  - line 159: should probably cite TabPFN\n  - line 195: broken reference ", "limitations": "- I think the fact that method requires dataset and feature descriptions is a limitation\n- Looking at the generated code, some features and their usefulness explanations are, almost surely, hallucinations. I think It's important to mention hallucinations in the limitations section to represent this. \n\n**References**\n- `[1]` Franziska Horn, Robert Pack, and Michael Rieger. The autofeat python library for automatic feature engineering and selection.\n- `[2]` James Max Kanter and Kalyan Veeramachaneni. Deep feature synthesis: Towards automating data\nscience endeavors\n- `[3]` Li, Liyao, et al. \"Learning a Data-Driven Policy Network for Pre-Training Automated Feature Engineering.\" \n- `[4]` Zhang, Tianping, et al. \"OpenFE: Automated Feature Generation beyond Expert-level Performance.\"", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- I find the idea of incorporating LLMs via code generation to tabular data automatic feature engineering clever and clean\n- Having interpretable features is a big plus and an advantage over prior automatic methods\n- If I'm not mistaken, the first positive showing of LLMs in tabular data (where using an LLM seem to improve the performance of an already strong tabular model)\n- Percussion taken to create a non-leaked test-set", "weaknesses": "The major issue (that's why the score is not as high as it could have been for me) is the lacking evaluation. Both in terms of baselines/complementary methods and datasets. \n- Authors compare and complement the proposed method with AutoFeat `[1]` and DFS `[2]`, while much more effective automatic feature engineering methods like FETCH `[3]` and OpenFE `[4]` exist. \n- Also, in the above-mentioned papers, the evaluations are much more extensive, including more datasets (also regression datasets) and base models. Implementing CAAFE in those benchmarks would be a very big plus for the validation of the proposed method.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the time and effort taken by you to evaluate our paper. We've carefully considered each point raised and aim to address them comprehensively below.  We are especially happy that you appreciate the interpretability of our generated features. We have in the meantime received lots of feedback, that this point is going to be especially valuable to practitioners.\n\nWe would also appreciate you reading through the general reviewer feedback, which outlines the changes we made and might address open questions that go beyond what you have asked for.\n\nFor this year's NeurIPS revisions, it is not possible to upload a modified paper, but only a one-page rebuttal PDF. Hence, we've outlined the changes we'll implement in the final paper within our responses.\n\n\n[Weaknesses]\n\n*[1] \"Authors compare and complement the proposed method with AutoFeat [1] and DFS [2], while much more effective automatic feature engineering methods like FETCH [3] and OpenFE [4] exist.\"*\n\nWe evaluated more baselines: (1) AutoFE methods: FETCH and OpenFE (2) AutoML methods: Autosklearn and Autogluon (Which perform feature engineering as part of their pipeline. Here, the effect of additional AutoFE should be smaller and could be negative). You can find detailed results in Table 2 (one-page rebuttal PDF) in the PDF. TLDR: CAAFE (GPT-4) + TabPFN is strongest among all methods and adds performance to AutoML methods. Using TabPFN as a classifier, a critical difference diagram shows statistical significance of CAAFE to all baselines. For FETCH, we could only evaluate it optimized for Logistic Regression due to the large computational cost (up to 24h / dataset / seed).\n\n*[2] \"Also, in the above-mentioned papers, the evaluations are much more extensive, including more datasets (also regression datasets) and base models. Implementing CAAFE in those benchmarks would be a very big plus for the validation of the proposed method.\"*\n\nFor CAAFE we have a special set of requirements which makes its evaluation more challenging than previous approaches: (1) CAAFE requires meaningful dataset descriptions, which are not available for all of these previous datasets (2) as stated in our limitations, the cost of applying CAAFE rises linearly with the number of features in a dataset, thus we focus on datasets with less than 20 features.\nWe believe that contextual prediction problems, where a dataset is modelled with a description of its use case, will grow in significance as deep learning becomes more multimodal and language models are applied to more modalities. A larger benchmark of datasets with interesting and varied contextual information will be vital to evaluate these works, especially when comparing multiple contextual algorithms.\n\n[Questions]\n\n*[1] \"What do the bold entries mean in the table? The standard deviation values are pretty large, but results are bolded. Were stat tests used?\"*\n\nWe simply bolded entries with the largest mean values. Standard deviations are across data-splits and can thus be quite large. We added an explanation to the table describing which entries are lighted. Also, we use a statistical significance test, comparing AutoFE using a critical difference diagram, that employs a Wilcoxon test for statistical significance (taking into account multiple testing for multiple tested methods). See our answer to Weakness [1].\n\n*[2] \"Does performance saturate after 10 iterations? What is the reason for stopping there?\"*\n\nSupplement F shows the cost and performance improvement when running CAAFE for 1-10 iterations. We see that performance is rising even after 10 iterations. 10 was simply the parameter that we started with. We did, however, not evaluate more iterations since we ran out of budget and could not repeat all experiments with another iteration parameter.\n\n*[3] \"Do you update the prompt with the generated feature descriptions for new iterations (I think info is missing in the paper, but this seems important)?\"*\n\nThe prompt in each iteration contains the previously generated code and the performance after executing that feature engineering step. Thus, CAAFE can learn from previous code operations. This important information has, indeed been missing from our work, and we did include it in our new version. In line 141, we add:\n\"F: Any code generated by CAAFE in previous iterations, as well as ROC AUC and accuracy evaluated on the validation splits.\"\n\n*[4] \"Could you report some numerical quantification of features generated by CAAFE, like feature importance ranking?\"*\n\nYes surely - we had a plot prepared for this that shows the feature importance of multiple datasets. Unfortunately, due to the page limit (1 page) for figures in this rebuttal, we could not include these plots... We will include this plot in the final work. On average the mean importance of generated features in our trained random forests was 1.49 times that of the original features.\n\n*[5] Minor issues: line 159: should probably cite TabPFN, line 195: broken reference*\n\nThank you, we have addressed these issues :)\n\n[Limitations]\n\n*[1] Looking at the generated code, some features and their usefulness explanations are, almost surely, hallucinations. I think It's important to mention hallucinations in the limitations section to represent this.*\n\nWe added the following section to our conclusion:\nLLMs, at times, exhibit a phenomenon known as \"hallucinations.\", where models produce inaccurate or invented information. Within CAAFE, this might result in the generation of features and associated explanations that appear significant and are logically presented, even though they may not be grounded in reality. Such behaviour can be problematic, especially when individuals place trust in these systems for essential decision-making or research tasks. \n"}, {"review_id": "ycxKrdIEwA", "reviewer": "Reviewer_ndjC", "summary": "In this paper, the authors propose a feature engineering method that builds upon LLMs. Features are engineering in an iterative process of prompting the LLM to generate code for new features, evaluating the features with a ML model and generating a new prompt. Hence, the overall approach follows the common wrapper approach. In their empirical evaluation they find feature engineering via LLMs to outperform no feature engineering.", "questions": "1. How does CAAFE compare to the SOTA in automated feature engineering?\n2. What makes CAAFE semi-automated? As far as I understood, CAAFE only requires an input by the user once in the beginning to give a description of the dataset. If this is the reason for semi-automation are not every AutoML tools out there semi-automated?\n3. Do LLMs in CAAFE really leverage semantic information about the dataset?\n4. As the authors already state in their Broader Impact Statement there is a certain risk that biases contained in LLMs transfer to the engineered features and that CAAFE then builds features based on these biases. Should not every feature engineering approach based on LLMs directly incorporate mechanisms to prevent to leverage such biases? Especially since such biases might be very subtle.", "limitations": "The authors prove a broader impact statement estimating what effect and potential negative societal impact there might be.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- To the best of my knowledge, this is the first work to employ LLMs (GPT) for automating feature engineering.\n- The general idea seems to work out quite nicely and compare comeptitively to other feature engineering methods and preferably over engineering no features.\n- The paper is well written and easy to follow. The overall presentation is really excellent - maybe due to the use of ChatGPT for the editing?", "weaknesses": "Although the idea in general is quite interesting, I have several doubts regarding this work. The first and foremost doubt is about how to sort this work into the already existing literature and whether it really establishes a new state of the art. In particular, I am thinking about the following two works which have been missed also during the discussion of related works:\nLi, Liyao, et al. \"Learning a Data-Driven Policy Network for Pre-Training Automated Feature Engineering.\" The Eleventh International Conference on Learning Representations. 2022.\nZhang, Tianping, et al. \"OpenFE: Automated Feature Generation beyond Expert-level Performance.\" arXiv preprint arXiv:2211.12507 (2022).\n(https://openreview.net/forum?id=1H1irbEaGV)\nThis might be due to these works being rather recent. However, preprints of these papers have already been available for some time. I believe a comparison to these methods is inevitable to prove the real benefit of LLMs and whether the semantic information of feature names is really needed to come up with better features.\n\nAnother issue with the paper is that it claims that LLMs are taking advantage of the semantical information of a dataset, e.g., the names of the features etc. But this is not substantiated in the experiments nor visible in the explanations generated in the examples.\n\nMoreover, I am a little bit doubtful to what extent the explanations of an LLM can be meaningful when LLMs base the text generated on the likelihood of the next word. In particular, logical argumentation does not seem to be a strengths of GPT so far and the flaws in logical reasoning can be made easily visible. E.g. testing with a sentence like \"The [doctor/nurse] yelled at the [doctor/nurse] because she was late. Who was late?\" and asking for an explanation for the decision. Why should the explanations be more meaningful for datasets?\n\nSpeaking about the semantics that are supposedly extracted from feature names, it would have been interesting to see whether this information is indeed used, e.g., by blinding feature names. Neither in the main paper nor in the supplement I could find experiments that aim at answering these questions. However, these experiments lacking mean an unproved claim made in the paper.\n\nMinor:\n- Reference to table in line 195 broken.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the time and effort taken by you to evaluate our paper. We've carefully considered each point raised and aim to address them comprehensively below. Thank you for appreciating the paper presentation - to answer your question about GPT-4 use: We did indeed use GPT-4 to prepare the work, which has been especially useful in creating and refining Latex plots! We mention this in the Acknowledgements section.\n\nFor this year's NeurIPS revisions, it is not possible to upload a modified paper, but only a one-page rebuttal PDF. Hence, we've outlined the changes we'll implement in the final paper within our responses.\n\n### Weaknesses\n\n1. *You ask for further comparisons to FETCH and OpenFE.*\n\nWe evaluated more baselines: (1) We additionally compared to the AutoFE methods FETCH and OpenFE, and (2) we additionally used the AutoML methods Autosklearn and Autogluon as base classifiers, which perform feature engineering as part of their pipeline. You can find detailed results in Table 2 (one-page rebuttal PDF) in the PDF. TLDR: We find that one of the two CAAFE variants is the best feature engineering method for all base classifiers in both mean and mean rank AUROC, but for logistic regression. Additionally, we find that CAAFE (GPT-4) + TabPFN is the strongest among all methods. Using TabPFN as a classifier, a critical difference diagram shows a statistically significant edge of CAAFE over all baselines. For FETCH, we could only evaluate it optimized for Logistic Regression due to the large computational cost (up to 24h / dataset / seed).\n\n2. *\"Another issue with the paper is that it claims that LLMs are taking advantage of the semantic information of a dataset, e.g., the names of the features etc. But this is not substantiated in the experiments\"*\n\nWe performed an additional ablation study to test this hypothesis. We blind feature names and dataset descriptions, i.e. CAAFE is applied to the datasets without contextual information. We find a strong drop in performance from an average AUROC of 0.822 to 0.8 over all datasets for GPT-4. Please, see Table 1 in the rebuttal PDF for detailed results.\n\nLooking at the generated features very strongly highlights the usefulness of semantic information. Consider e.g. the tic-tac-toe dataset: The generated features are based on up to 9 features - a useful combination of that many features is very hard to find with random search but given contextual information this feature is quickly discovered.\n\n3. *\"Moreover, I am a little bit doubtful to what extent the explanations of an LLM can be meaningful when LLMs base the text generated on the likelihood of the next word. [..]\"*\n\nGenerated explanations do not need to be complex, in many cases this can be a simple explanation of what the generated code does. This can already be quite useful to non-technical users.\n\nWe published all generated features for you to inspect at https://github.com/cafeautomatedfeatures/CAFE/tree/main/data/generated_code. They show that generated feature explanations are often of high quality and include semantic information. This quality of the provided explanations often surprised us as well. We have in the meantime received a significant amount of feedback that highlights that this interpretability is going to be especially valuable to practitioners. Take the following generation for balance_scale as an example: https://github.com/cafeautomatedfeatures/CAFE/blob/main/data/generated_code/balance-scale_v4_0_code.txt\nHere the task is predicting if a scale tips.\n\nWe do, however, want to address the issue of false explanations and to our limitations:\nLLMs, at times, exhibit a phenomenon known as \"hallucinations\", where models produce inaccurate or invented information. Within CAAFE, this might result in the generation of features and associated explanations that appear significant and are logically presented, even though they may not be grounded in reality. Such behavior can be problematic, especially when individuals place trust in these systems for essential decision-making or research tasks.\n\n### Questions\n\n1. *How does CAAFE compare to the SOTA in automated feature engineering?*\n\nWe believe an answer is contained in our reply to your first point of weaknesses.\n\n2. *What makes CAAFE semi-automated? As far as I understood, CAAFE only requires an input by the user once in the beginning to give a description of the dataset. If this is the reason for semi-automation are not every AutoML tools out there semi-automated?$\n\nOur intention was the following: Classical AutoML deals with model selection and technical optimizations which do not require semantic information. The term AutoDS or Automated Data Science often refers to a more extensive automation of the Data Science stack, where AutoML just covers a portion. We chose the term semi-automated Data Science since we are not automating all of the data science stack, but another part of it, while highlighting that this is a step towards AutoDS, i.e. full automation of the DS pipeline. The naming might be confusing, however, and we are considering a slight change.\n\n3. *Do LLMs in CAAFE really leverage semantic information about the dataset?*\n\nSee the reply to Weakness 2.\n\n4. *As the authors already state in their Broader Impact Statement there is a certain risk that biases contained in LLMs transfer to the engineered features and that CAAFE then builds features based on these biases. Should not every feature engineering approach based on LLMs directly incorporate mechanisms to prevent to leverage such biases? Especially since such biases might be very subtle.*\n\nWe have formulated a detailed answer in our reply to the ethics reviewer Ce9G and refer you there for an in-depth discussion of biases as we only have limited space in each rebuttal post."}, {"review_id": "2lgI2UbKrL", "reviewer": "Reviewer_Cmyk", "summary": "This paper presents Context-Aware Automated Feature Engineering (CAAFE) for integrating domain knowledge into the AutoML process using LLMs. CAAFE automates feature engineering for tabular datasets, generating Python code that generates semantically meaningful features based on the dataset and a textual description of the data.  The study demonstrates the effectiveness of CAAFE on a range of benchmark tasks.\n\n", "questions": "- What about the dataset where performance does not improve or even weaken? Is there an explanation.", "limitations": "- The main limitation is more extensive study of competing approaches. In essence the authors work really doesn't answer any open scientific questions.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "- Well written and easy to follow. Thank you!\n- Simple and very applicable to real-world scenarios!", "weaknesses": "- If set up correctly, the method really can't fail. Features are only added, if they improve the objective. Thus the scientific value is somewhat limited to showing that LLMs can create features that improve performance (expected) and the extent to which (here is where I see most value). Therefore comparison to other feature engineering methods seems really important and could be more extensive.\n- Fig 2 is not clear. Is this a flow from left to right? How does the bottom part relate to the boxes? Please explain better. \n\nSuggestions:\n- I don't think the phrase \"code as interface\" is self-explanatory. Maybe can be introduced explicitly\n- In section 2.1., the sentence \"GPT-4 is a deep neural network that uses a transformer architecture\" - may not actually be correct. It is rumoured that GPT4 consists of multiple models and uses some mixture of experts methodology. In any case, this information is not published which might be worthwhile pointing out.\n- l195, table reference is missing", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the time and effort taken by you to evaluate our paper. We've carefully considered each point raised and aim to address them comprehensively below. Thank you for appreciating the paper presentation!\n\nWe would like to address your questions and our changes in response to them in detail below. We would also appreciate you reading through the general reviewer feedback, which outlines the changes we made and might address open questions that go beyond what you have asked for.\n\nFor this years NeurIPS revisions, it is not possible to upload a modified paper, but only upload a one page rebuttal with additional figures. When changes in text of the main paper were made, we replied these changes to each reviewer individually.\n\nWe were a bit surprised at the low contribution score - we believe this paradigm of introducing context / semantic into automated machine learning can be quite meaningful and it is highly original (Reviewer uCcq: \"​​Although the seminal idea proposed in the paper is rather straightforward, it's novelty makes it a worthwhile body of research for consideration.\", Reviewer Y55o: \"If I'm not mistaken, the first positive showing of LLMs in tabular data\", viewer ndjC: \"To the best of my knowledge, this is the first work to employ LLMs (GPT) for automating feature engineering.\")\n\n[Weaknesses]\n\n*[1] \"If set up correctly, the method really can't fail. Features are only added, if they improve the objective. Thus the scientific value is somewhat limited to showing that LLMs can create features that improve performance (expected) and the extent to which (here is where I see most value).\"*\n\nA featurization is only validated on the training data, but performance is measured on separate test data. Overfitting can occur in the same way as it does in training a model - a featurization can be viewed as just another step in the model construction. When the number of provided samples is small or the number of tested featurizations is large, the risk of overfitting becomes larger. By considering semantically meaningful features CAAFE contains a prior for features that are more likely to generalize to the test set. Also, it reduces computational complexity by considering useful features more quickly. Looking at the generated code, these featurizations can contain up to 9 features at the same time - imagine the likelihood of such a feature when randomly permuting.\n\n*[2] \"Therefore comparison to other feature engineering methods seems really important and could be more extensive.\"*\n\nWe evaluated more baselines: (1) AutoFE methods: FETCH and OpenFE (2) AutoML methods: Autosklearn and Autogluon (Which perform feature engineering as part of their pipeline. Here, the effect of additional AutoFE should be smaller and could be negative). You can find detailed results in Table 2 (one-page rebuttal PDF) in the PDF. TLDR: CAAFE (GPT-4) + TabPFN is strongest among all methods and adds performance to AutoML methods. Using TabPFN as a classifier, a critical difference diagram shows statistical significance of CAAFE to all baselines. For FETCH, we could only evaluate it optimized for Logistic Regression due to the large computational cost (up to 24h / dataset / seed).\n\n*[3] Fig 2 is not clear. Is this a flow from left to right? How does the bottom part relate to the boxes? Please explain better.*\n\nThank you for the feedback! We removed the arrows and adapted the caption: \"Figure 2: Data Science pipeline, inspired by De Bie et al. (2022). CAAFE allows for automation of semantic data engineering, while LLMs could provide even further automation: (1) Context specification is user driven (2) exploitation and data engineering can be automated through LLMs (3) model building can be automated by classical AutoML approaches\". We hope this would make this diagram clearer.\n\n[Suggestions]\n\n*[1] I don't think the phrase \"code as interface\" is self-explanatory. Maybe can be introduced explicitly*\n\nWe did so in our updated manuscript, thank you! 🙂\n\"We bridge the gap between LLMs and classical algorithms by using code as an interface between them: LLMs generate code that modifies input datasets, these modified datasets can then be processed by classical algorithms.\"\n\n*[2] In section 2.1., the sentence \"GPT-4 is a deep neural network that uses a transformer architecture\" - may not actually be correct. It is rumoured that GPT4 consists of multiple models and uses some mixture of experts methodology. In any case, this information is not published which might be worthwhile pointing out.*\n\nWe did so in our updated manuscript, we write: \"The architecture of GPT-4 is not published, it is likely based on a deep neural network that uses a transformer architecture [...]\"\n\n*[3] l195, table reference is missing*\n\nThank you, done\n\n[Questions]\n\n*[1] What about the dataset where performance does not improve or even weaken? Is there an explanation.*\n\nWe believe an answer to this is contained in our reply to your first point of weaknesses.\n[Limitations]\n\n*[2] The main limitation is more extensive study of competing approaches.*\n\nWe believe an answer to this is contained in our reply to your second point of weaknesses."}, {"review_id": "cwkCdb6881", "reviewer": "Reviewer_uCcq", "summary": "The paper proposes CAAFE, a Context aware automated feature engineering approach to support auto ML by utilizing Large Language Models (LLMs). Proposed method generates new features or transforming the feature space from tabular data by using LLMs and incorporate them to train new models. Transforming the feature space can be an iterative process where in each step LLM generates code that is run on training and evaluation sets, followed by fitting a new model and evaluating its performance. LLM receives a set of instructions through a prompt which includes semantic information about the data set, feature names, their data types, useful stats about features, and a few sample values.\nThe paper reports experimental results on two sets of data sets: 1- OpenML data sets published before Sept 2021 (GPT 3.5 and 4 training date); 2- Kaggle data sets published after Sept 2021. Results show that CAAFE can improve the performance of a state of the art classifier. They also show that CAAFE can help build models that outperform models built with the help of other state of the art automatic feature generation.", "questions": "- How do you make sure the generated feature makes sense? e.g. https://github.com/cafeautomatedfeatures/CAFE/blob/main/data/generated_code/airlines_v3_0_code.txt why Euclidean distance between airports even make sense? Based on the data set description it is not clear if from or to airport attributes represent distance to a canonical point.\n- Minor, line 195, please fix the question marks in \"Table ??\".", "limitations": "The main limitation of the work is stated by authors as \"We focus on small datasets with up to 2,000 samples in total, because feature engineering is most important and significant for smaller datasets\". As stated, the proposed approach is mainly beneficial for smaller data sets which makes the method less useful for a lot of real world and specially industrial applications.\nAlong the same lines, as clearly mentioned by author(s), another limit of the work is the possibility of exceeding the prompt maximum length.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Although the seminal idea proposed in the paper is rather straightforward, it's novelty makes it a worthwhile body of research for consideration.\n\n- originality: The work is novel and original. It introduces the use of LLMs for auto ML through adding contextual information which is less explored.\n- quality: The paper is technically sound and experimental results support the claims. Limitation and risks of the work are also clearly called out.\n- clarity: The paper is easy to read and follow and the contributions are discussed clearly.\n- significance: Experimental results show LLM's usefulness for feature engineering aspect of AutoML which testifies the significance of the work. For more detailed comments on submission's significance refer to the limitation section.", "weaknesses": "- Alternative feature generation methods that are considered in the experiments are limited. AutoFE is the main alternative method. Other techniques such as PCA or deep sparse feature learning (authors have mentioned that with small data sets they don't expect much benefit from deep learning methods but it would be interesting to emperically validate this hypothesis for the studied data sets) is not considered.\n- The human-in-the-loop aspect of CAAFE is not fully discussed. For example, one of the implications of having human in the loop is correcting the potential mistakes that LLM may make. In the paper it is not discussed how many times the generated code by LLM needed to be revised by human, not necessarily due to run time errors but because of the semantic issues in the generated features.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the time and effort taken by you to evaluate our paper. We've carefully considered each point raised and aim to address them comprehensively below.\n\nFor this year's NeurIPS revisions, it is not possible to upload a modified paper, but only a one-page rebuttal PDF. Hence, we've outlined the changes we'll implement in the final paper within our responses.\n\n[Weaknesses]\n\n*[1] Alternative feature generation methods that are considered in the experiments are limited. AutoFE is the main alternative method. Other techniques such as PCA or deep sparse feature learning (authors have mentioned that with small data sets they don't expect much benefit from deep learning methods but it would be interesting to emperically validate this hypothesis for the studied data sets) is not considered.*\n\nWe evaluated more baselines: (1) AutoFE methods: FETCH and OpenFE (2) AutoML methods: Autosklearn and Autogluon (Which perform feature engineering as part of their pipeline. Here, the effect of additional AutoFE should be smaller and could be negative). Autosklearn e.g. also includes PCA among others for feature engineering. Please see the first point in our general rebuttal, as well as Figure 2 in the rebuttal PDF for detailed results You can find detailed results in Table 2 (one-page rebuttal PDF) in the PDF. TLDR: CAAFE (GPT-4) + TabPFN is strongest among all methods and adds performance to AutoML methods. Using TabPFN as a classifier, a critical difference diagram shows statistical significance of CAAFE to all baselines. For FETCH, we could only evaluate it optimized for Logistic Regression due to the large computational cost (up to 24h / dataset / seed)..\n\n*[2] The human-in-the-loop aspect of CAAFE is not fully discussed. For example, one of the implications of having human in the loop is correcting the potential mistakes that LLM may make. In the paper it is not discussed how many times the generated code by LLM needed to be revised by human, not necessarily due to run time errors but because of the semantic issues in the generated features.*\n\nWe mention that human-in-the-loop AutoML could be possible since semantic explanations can be more interpretable. An empirical evaluation of a human-in-the-loop concept would require running experiments with a cohort of users interacting with our algorithm. With our resources this is not feasible in terms of cost and time. This could be a followup to our work, likely coming from the industry, if commercialization is a goal.\n\n[Questions]\n\n*[1] How do you make sure the generated feature makes sense? e.g. https://github.com/cafeautomatedfeatures/CAFE/blob/main/data/generated_code/airlines_v3_0_code.txt why Euclidean distance between airports even make sense? Based on the data set description it is not clear if from or to airport attributes represent distance to a canonical point.*\n\nThis file has been generated by GPT-3.5 which produces less meaningful, coherent and empirically useful features (All files with \"v3\" in the name are generated by GPT-3.5, \"v4\" is GPT-4 - which we did not mention!). Evaluating if generated features are semantically meaningful is hard, since such an evaluation itself would be subjective. Once again, it would require a cohort of raters, judging subjectively. We do report examples in our paper and in the supplements and address in our limitations that there is no guarantee for features to be meaningful - however, looking at a few examples you quickly see that many are - while not perfect this is a significant leap forward to classical feature engineering.\n\nWe added the following to our limitations:\n\"LLMs, at times, exhibit a phenomenon known as \"hallucinations.\", where models produce inaccurate or invented information. Within CAAFE, this might result in the generation of features and associated explanations that appear significant and are logically presented, even though they may not be grounded in reality. Such behavior can be problematic, especially when individuals place trust in these systems for essential decision-making or research tasks.\"\n\n*[2] Minor, line 195, please fix the question marks in \"Table ??\".*\n\nDone, thank you! :)"}]

Author rebuttal:
Thank you very much for your constructive feedback. We believe that the reviews have helped us to improve our work significantly. For this year's NeurIPS review, it is not possible to upload a revised paper, only a one-page rebuttal PDF. Hence, we've detailed the changes we'll be making in the final paper in our responses.

### Further evaluation of baselines

We've expanded our evaluation to include

i. AutoFE methods: FETCH and OpenFE.

ii. AutoML methods: Autosklearn and Autogluon, which perform feature engineering as part of their pipeline.

You can find detailed results in Table 2 in the rebuttal PDF, and provide a summary here:

1. We find that among all feature engineering baselines and classifiers, the strongest combination is CAAFE (GPT-4) with TabPFN in terms of mean and mean rank AUROC. In addition, we find that CAAFE is the best feature engineering method for all base classifiers in both mean and mean rank AUROC, but for logistic regression.

2. We construct a critical difference plot that performs statistical tests between the feature engineering baselines. CAAFE (GPT-4) performs statistically significantly better than the baselines when using TabPFN, the strongest base classifier. Critical difference plots perform a Wilcoxon signed rank test with correction for multiple testing.

3. Notably, CAAFE improves the performance of all methods, even AutoML methods with built-in AutoFE. This underscores CAAFE's unique ability to integrate semantic/contextual information missing in traditional feature engineering.

4. CAAFE can be combined with AutoFE baseline methods and AutoML, see Table 2 in our original manuscript. CAAFE combined with an AutoFE baseline is the strongest predictor for simple classifiers (logistic regression and random forest). Here, the strength of CAAFE (context integration) is added to the strength of the baselines (combination of a large number of features). For AutoML methods that already perform AutoFE, we do not see this effect, and only the use of CAAFE is strongest.

As stated in our work, we do not consider CAAFE to work in the same domain as the baselines. CAAFE performs the novel semantic/contextual feature engineering and obtains additional information, while baselines perform data transformation based solely on the data matrix.

### Integration of Semantic Information

1. We conducted an experiment blinding the semantic information to more explicitly show the influence of semantic information on CAAFE's performance. By excluding feature names and dataset descriptions, we found a significant drop in performance, from an average AUROC of 0.822 to 0.8. Details can be found in Table 1 of our rebuttal PDF. See Table 1 (one-page rebuttal PDF) for details.

2. Looking at the generated features in https://github.com/cafeautomatedfeatures/CAFE/tree/main/data/generated_code gives a good intuition about the value of semantic information (files with V4 were generated by GPT-4, V3 by GPT-3.5): Some of these generated features contain up to 9 variables combined in a meaningful way. Generating such combinations without a strong semantic prior of meaningful combinations would be extremely expensive. Also note that a feature is only validated on the training data, but performance is measured on separate test data. Overfitting can occur in the same way as when training a model - a feature extraction can be seen as just another step in model construction. When the number of samples provided is small, or the number of features tested is large, the risk of overfitting becomes greater. By considering semantically meaningful features, CAAFE includes a prior for features that are more likely to generalize to the test set.


Meta-review: Using LLM for automating feature construction based on the available feature names is a simple and brilliant (engineering) idea.

This paper has the merit to implement this idea and conduct the experiments with i) extensive awareness of the ethical risks; ii) extensive test on datasets from OpenML and Kaggle (arguing that the latter are less prone to be viewed by LLM). 

The reviewers ask for extensive additional evidence and the authors conducted experiments and comparisons with other feature construction approaches.

Although one might always ask for more experiments, the AC thinks that there is enough in the paper to give food for thought for the NeurIPS audience; detailing and discussing the methodology deployed to mitigate the ethical risks would be particularly useful.
Decision: Accept (poster)