Title: Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering
Venue: NeurIPS 2023
Accepted: yes
Keywords: AutoML

AutoDS

Automated Feature Engineering

LLM Code Generation

Tabular Data

Feature Engineering

Automated Data Science

Automated Machine Learning
Abstract: As the field of automated machine learning (AutoML) advances, it becomes increasingly important to incorporate domain knowledge into these systems.
We present an approach for doing so by harnessing the power of large language models (LLMs). Specifically, we introduce Context-Aware Automated Feature Engineering (CAAFE), a feature engineering method for tabular datasets that utilizes an LLM to iteratively generate additional semantically meaningful features for tabular datasets based on the description of the dataset. The method produces both Python code for creating new features and explanations for the utility of the generated features.

Despite being methodologically simple, CAAFE improves performance on 11 out of 14 datasets -- boosting mean ROC AUC performance from 0.798 to 0.822 across all dataset - similar to the improvement achieved by using a random forest instead of logistic regression on our datasets. Furthermore, CAAFE is interpretable by providing a textual explanation for each generated feature.

CAAFE paves the way for more extensive semi-automation in data science tasks and emphasizes the significance of context-aware solutions that can extend the scope of AutoML systems to semantic AutoML. We release our [code](url{https://github.com/automl/CAAFE), a simple [demo](url{https://colab.research.google.com/drive/1mCA8xOAJZ4MaB_alZvyARTMjhl6RZf0a) and a [python package](url{https://pypi.org/project/caafe/).

Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering

Abstract

As the field of automated machine learning (AutoML) advances, it becomes increasingly important to incorporate domain knowledge into these systems. We present an approach for doing so by harnessing the power of large language models (LLMs). Specifically, we introduce Context-Aware Automated Feature Engineering (CAAFE), a feature engineering method for tabular datasets that utilizes an LLM to iteratively generate additional semantically meaningful features for tabular datasets based on the description of the dataset. The method produces both Python code for creating new features and explanations for the utility of the generated features.

Despite being methodologically simple, CAAFE improves performance on 11 out of 14 datasets - boosting mean ROC AUC performance from 0.798 to 0.822 across all dataset - similar to the improvement achieved by using a random forest instead of logistic regression on our datasets.

Furthermore, CAAFE is interpretable by providing a textual explanation for each generated feature. CAAFE paves the way for more extensive semi-automation in data science tasks and emphasizes the significance of context-aware solutions that can extend the scope of AutoML systems to semantic AutoML. We release our code, a simple demo and a python package.

CAAFE

User:

Specifies

problem context

and dataset

LLM:

Generates

Code for feature

engineering

Interpreter:

Executes

generated code

Tabular Pre-

diction Model:

Performs

cross-validation.

Evaluate Performance.

Keep change if performance is improved.

Frank Hutter

University of Freiburg Prior Labs fh@cs.uni-freiburg.de

1 Introduction

Automated machine learning (AutoML; e.g., Hutter et al. (2019)) is very effective at optimizing the machine learning (ML) part of the data science workflow, but existing systems leave tasks such as data engineering and integration of domain knowledge largely to human practitioners. However, model selection, training, and scoring only account for a small percentage of the time spent by data scientists (roughly 23% according to the 'State of Data Science'(Anaconda, 2020)). Thus, the most time-consuming tasks, namely data engineering and data cleaning, are only supported to a very limited degree by AutoML tools, if at all.

While the traditional AutoML approach has been fruitful and appropriate given the technical capabilities of ML tools at the time, large language models (LLMs) may extend the reach of AutoML to cover more of data science and allow it to evolve towards automated data science (De Bie et al., 2022). LLMs encapsulate extensive domain knowledge that can be used to automate various data science tasks, including those that require contextual information. They are, however, not interpretable, or verifiable, and behave less consistently than classical ML algorithms. E.g., even the best LLMs still fail to count or perform simple calculations that are easily solved by classical methods (Hendrycks et al., 2021; OpenAI Community, 2021).

In this work, we propose an approach that combines the scalability and robustness of classical ML classifiers (e.g. random forests (Breiman, 2001)) with the vast domain knowledge embedded in LLMs, as visualized in Figure 2. We bridge the gap between LLMs and classical algorithms by using code as an interface between them: LLMs generate code that modifies input datasets, these modified datasets can then be processed by classical algorithms. Our proposed method, CAAFE, generates Python code that creates semantically meaningful features that improve the performance of downstream prediction tasks in an iterative fashion and with algorithmic feedback as shown in Figure 1. Furthermore, CAAFE generates a comment for each feature which explains the utility of generated feature. This allows interpretable AutoML, making it easier for the user to understand a solution, but also to modify and improve on it. Our approach combines the advantages of classical ML (robustness, predictability and a level of interpretability) and LLMs (domain-knowledge and creativity).

Automating the integration of domain-knowledge into the AutoML process has clear advantages that extend the scope of existing AutoML methods. These benefits include: i) Reducing the latency from data to trained models; ii) Reducing the cost of creating ML models; iii) Evaluating a more informed space of solutions than previously possible with AutoML, but a larger space than previously possible with manual approaches for integrating domain knowledge; and iv) Enhancing the robustness and reproducibility of solutions, as computer-generated solutions are more easily reproduced. CAAFE demonstrates the potential of LLMs for automating a broader range of data science tasks and highlights the emerging potential for creating more robust and context-aware AutoML tools.

2 Background

2.1 Large Language Models (LLMs)

LLMs are neural networks that are pre-trained on large quantities of raw text data to predict the next word in text documents. Recently, GPT-4 has been released as a powerful and publicly available LLM (OpenAI, 2023a). The architecture of GPT-4 is not oublished, it is likely based on a deep neural network that uses a transformer architecture (Vaswani et al., 2017), large-scale pre-training on a diverse corpus of text and fine-tuning using reinforcement learning from human feedback (RLHF) (Ziegler et al., 2019). It achieves state-of-the-art performance on various tasks, such as text generation, summarization, question answering and coding. One can adapt LLMs to a specific task without retraining by writing a prompt (Brown et al., 2020; Wei et al., 2021); the model parameters are frozen and the model performs in-context inference tasks based on a textual input that formulates the task and potentially contains examples.

LLMs as Tabular Prediction Models Hegselmann et al. (2023) recently showed how to use LLMs for tabular data prediction by applying them to a textual representation of these datasets. A prediction on an unseen sample then involves continuing the textual description of that sample on the target column. However, this method requires encoding the entire training dataset as a string and processing it using a transformer-based architecture, where the computational cost increases

Context specification

contextual informa-

tion, goal analysis

and specification

Exploitation

interpretation, visual-

ization, reporting, pre-

dictions, monitoring

Data Engineering

wrangling, integra-

tion, preparation,

transformation

algorithm choice,

ing, evaluation,

model selection

CAAFE

Traditional AutoML

User driven

Strength of LLMs: Exploratory, context-dependent, real-world knowledge

Strength of classical algorithms: Well-defined, predictable

quadratically with respect to N · M , where N denotes the number of samples and M the number of features. Furthermore, the predictions generated by LLMs are not easily interpretable, and there is no assurance that the LLMs will produce consistent predictions, as these predictions depend directly on the complex and heterogeneous data used to train the models. So far, Hegselmann et al. (2023) found that their method yielded the best performance on tiny datasets with up to 8 samples, but was outperformed for larger data sets.

LLMs for Data Wrangling Narayan et al. (2022) demonstrated state-of-the-art results using LLMs for entity matching, error detection, and data imputation using prompting and manually tuning the LLMs. Vos et al. (2022) extended this technique by employing an improved prefix tuning technique. Both approaches generate and utilize the LLMs output for each individual data sample, executing a prompt for each row. This is in contrast to CAAFE, which uses code as an interface, making our work much more scalable and faster to execute, since one LLM query can be applied to all samples.

2.2 Feature Engineering

Feature engineering refers to the process of constructing suitable features from raw input data, which can lead to improved predictive performance. Given a dataset D = ( x i , y i ) n i =1 , the goal is to find a function ϕ : X → X ′ which maximizes the performance of A ( ϕ ( x i ) , y i ) for some learning algorithm A . Common methods include numerical transformations, categorical encoding, clustering, group aggregation, and dimensionality reduction techniques, such as principal component analysis (Wold et al., 1987).

Deep learning methods are capable of learning suitable transformations from the raw input data making them more data-driven and making explicit feature engineering less critical, but only given a lot of data. Thus, appropriate feature engineering still improves the performance of classical and deep learning models, particularly for limited data, complex patterns, or model interpretability.

Various strategies for automated feature engineering have been explored in prior studies. Deep Feature Synthesis (DFS; Kanter & Veeramachaneni (2015)) integrates multiple tables for feature engineering by enumerating potential transformations on features and performing feature selection based on model performance. Cognito (Khurana et al., 2016) proposes a tree-like exploration of the feature space using handcrafted heuristic traversal strategies. AutoFeat (Horn et al., 2019) employs an iterative subsampling of features using beam search. Learning-based methods, such as LFE (Nargesian et al., 2017), utilize machine learning models to recommend beneficial transformations while other methods use reinforcement learning-based strategies (Khurana et al., 2018; Zhang et al., 2019). Despite these advancements, none of the existing methods can harness semantic information in an automated manner.

2.2.1 Incorporating Semantic Information

1

2

3

4

5

6

7

8

9 1011121314

Initial data

True

False

Test-Sample

1

2

3

4

5

6

7

8

9 1011121314

Weekday

Weekend

With extra feature based on context

Working

Day-Off

Test-Sample

The potential feature space, when considering the combinatorial number of transformations and combinations, is vast. Therefore, semantic information is useful, to serve as a prior for identifying useful features. By incorporating semantic and contextual information, feature engineering techniques can be limited to semantically meaningful features enhancing the performance by mitigating issues with multiple testing and computational complexity and boosting the interpretability of machine learning models. This strategy is naturally applied by human experts who leverage their domainspecific knowledge and insights. Figure 3 exemplifies the usefulness of contextual information.

3 Method

We present CAAFE, an approach that leverages large language models to incorporate domain knowledge into the feature engineering process, offering a promising direction for automating data science tasks while maintaining interpretability and performance.

Our method takes the training and validation datasets, D train and D valid , as well as a description of the context of the training dataset and features as input. From this information CAAFE constructs a prompt, i.e. instructions to the LLM containing specifics of the dataset and the feature engineering task. Our method performs multiple iterations of feature alterations and evaluations on the validation dataset, as outlined in Figure 1. In each iteration, the LLM generates code, which is then executed on the current D train and D valid resulting in the transformed datasets D ′ train and D ′ valid . We then use D ′ train to fit an ML-classifier and evaluate its performance P ′ on D ′ valid . If P ′ exceeds the performance P achieved by training on D train and evaluating on D valid , the feature is kept and we set D train := D ′ train and D valid := D ′ valid . Otherwise, the feature is rejected and D train and D valid remain unchanged. Figure 4 shows a shortened version of one such run on the Tic-Tac-Toe Endgame dataset.

Prompting LLMs for Feature Engineering Code Here, we describe how CAAFE builds the prompt that is used to perform feature engineering. In this prompt, the LLM is instructed to create valuable features for a subsequent prediction task and to provide justifications for the added feature's utility. It is also instructed to drop unnecessary features, e.g. when their information is captured by other created features.

The prompt contains semantic and descriptive information about the dataset. Descriptive information, i.e. summary statistics, such as the percentage of missing values is based solely on the train split of the dataset. The prompt consists of the following data points:

A A user-generated dataset description, that contains contextual information about the dataset (see Section 4 for details on dataset descriptions for our experiments)

B Feature names adding contextual information and allowing the LLM to generate code to index features by their names

Figure 4: Exemplary run of CAAFE on the Tic-Tac-Toe Endgame dataset. User generated input is shown in blue, ML-classifier generated data shown in red and LLM generated code is shown with syntax highlighting. The generated code contains a comment per generated feature that follows a template provided in our prompt (Feature name, description of usefulness, features used in the generated code and sample values of these features). In this run, CAAFE improves the ROC AUC on the validation dataset from 0.888 to 1.0 in two feature engineering iterations.

C Data types (e.g. float, int, category, string) - this adds information on how to handle a feature in the generated code

D Percentage of missing values - missing values are an additional challenge for code generation

E 10 random rows from the dataset - this provides information on the feature scale, encoding, etc.

Additionally, the prompt provides a template for the expected form of the generated code and explanations. Adding a template when prompting is a common technique to improve the quality of responses (OpenAI, 2023b). We use Chain-of-thought instructions - instructing a series of intermediate reasoning steps -, another effective technique for prompting (Wei et al., 2023). The prompt includes an example of one such Chain-of-thought for the code generation of one feature: first providing the high-level meaning and usefulness of the generated feature, providing the names of features used to generate it, retrieving sample values it would need to accept and finally writing a line of code. We provide the complete prompt in Figure 5 in the appendix.

If the execution of a code block raises an error, this error is passed to the LLM for the next code generation iteration. We observe that using this technique CAAFE recovered from all errors in our experiments. One such example can be found in Table 3.

Technical Setup The data is stored in a Pandas dataframe (Wes McKinney, 2010), which is preloaded into memory for code execution. The generated Python code is executed in an environment where the training and validation data frame is preloaded. The performance is measured on the current dataset with ten random validation splits D valid and the respective transformed datasets D ′ valid with the mean change of accuracy and ROC AUC used to determine if the changes of a code block are kept, i.e. when the average of both is greater than 0. We use OpenAI's GPT-4 and GPT-3.5 as LLMs (OpenAI, 2023a) in CAAFE. We perform ten feature engineering iterations and TabPFN (Hollmann et al., 2022) in the iterative evaluation of code blocks.

The automatic execution of AI-generated code carries inherent risks, such as misuse by malicious actors or unintended consequences from AI systems operating outside of controlled environments. Our approach is informed by previous studies on AI code generation and cybersecurity (Rohlf, 2023; Crockett, 2023). We parse the syntax of the generated python code and use a whitelist of operations that are allowed for execution. Thus operations such as imports, arbitrary function calls and others are excluded. This does not provide full security, however, e.g. does not exclude operations that can lead to infinite loops and excessive resource usage such as loops and list comprehensions.

4 Experimental Setup

Setup of Downstream-Classifiers We evaluate our method with Logistic Regression, Random Forests (Breiman, 2001) and TabPFN (Hollmann et al., 2022) for the final evaluation while using TabPFN to evaluate the performance of added features. We impute missing values with the mean, one-hot or ordinal encoded categorical inputs, normalized features and passed categorical feature indicators, where necessary, using the setup of Hollmann et al. (2022) 1 .

Setup of Automated Feature Engineering Methods We also evaluate popular context-agnostic feature engineering libraries Deep Feature Synthesis (DFS; Kanter & Veeramachaneni, 2015) and AutoFeat (Horn et al., 2019) 2 . We evaluate DFS and AutoFeat alone and in combination with CAAFE. When combined, CAAFE is applied first and the context-agnostic AutoFE method subsequently. For DFS we use the primitives "add_numeric" and "multiply_numeric", and default settings otherwise. For TabPFN, DFS generates more features than TabPFN accepts (the maximum number of features is 100) in some cases. In these cases, we set the performance to the performance without feature engineering. For AutoFeat, we use one feature engineering step and default settings otherwise.

Evaluating LLMs on Tabular Data The LLM's training data originates from the web, potentially including datasets and related notebooks. GPT-4 and GPT-3.5 have a knowledge cutoff in September 2021, i.e., almost all of its training data originated from before this date. Thus, an evaluation on established benchmarks can be biased since a textual description of these benchmarks might have been used in the training of the LLM.

We use two categories of datasets for our evaluation: (1) widely recognized datasets from OpenML released before September 2021, that could potentially be part of the LLMs training corpus and (2) lesser known datasets from Kaggle released after September 2021 and only accessible after accepting an agreement and thus harder to access by web crawlers.

From OpenML (Vanschoren et al., 2013; Feurer et al.), we use small datasets that have descriptive feature names (i.e. we do not include any datasets with numbered feature names). Datasets on OpenML contain a task description that we provide as user context to our method. When datasets are perfectly solvable with TabPFN alone (i.e. reaches ROC AUC of 1.0) we reduce the training set size for that dataset, marked in Table 1. We focus on small datasets with up to 2 000 samples in total, because feature engineering is most important and significant for smaller datasets.

We describe the collection and preprocessing of datasets in detail in Appendix G.1.

Evaluation Protocol For each dataset, we evaluate 5 repetitions, each with a different random seed and train- and test split to reduce the variance stemming from these splits (Bouthillier et al., 2021). We split into 50% train and 50% test samples and all methods used the same splits.

5 Results

In this section we showcase the results of our method in three different ways. First, we show that CAAFE can improve the performance of a state-of-the-art classifier. Next, we show how CAAFE interacts with traditional automatic feature engineering methods and conclude with examples of the features that CAAFE creates.

Table 2: Mean ROC AUC and average rank (ROC AUC) per downstream classification method and feature extension method. Best AutoFE method per base classifer is shown in bold. The features generated by CAAFE are chosen with TabPFN as classifier. Rank are calculated across all classifiers and feature engineering methods. FETCH was too computationally expensive to compute for all base classifiers in the rebuttal. Each seed and dataset takes up to 24 hours and has to be evaluated for each base classifer independently. Thus, we use features computed for logistic regression for all other classifiers.

Performance of CAAFE CAAFE can improve our strongest classifier, TabPFN, substantially. If it is used with GPT-4, we improve average ROC AUC performance from 0.798 to 0.822, as shown in Table 2, and enhance the performance for 11/14 datasets.On the evaluated datasets, this improvement

is similar (71%) to the average improvement achieved by using a random forest (AUC 0.783) instead of logistic regression (AUC 0.749). We can see that CAAFE even improves performance for all of the new datasets from Kaggle. If we use CAAFE with GPT-3.5 only, we can see that it performs clearly worse than with GPT-4, and only improves performance on 6/14 datasets.

There is great variability in the improvement size depending on whether (1) a problem is amenable to feature engineering, i.e. is there a mapping of features that explains the data better and that can be expressed through simple code; and (2) the quality of the dataset description (e.g., the balance-scale dataset contains an accurate description of how the dataset was constructed) Per dataset performance can be found in Table 1. CAAFE takes 4:43 minutes to run on each dataset, 90% of the time is spent on the LLM's code generation and 10% on the evaluation of the generated features. In Appendix F we plot the performance, time and cost of CAAFE across feature engineering iterations, showing the tradeoff between these parameters. For the 14 datasets, 5 splits and 10 CAAFE iterations, CAAFE generates 52 faulty features (7.4%) in the generation stage, from which it recovers (see Figure 3).

Incorporating Classical AutoFE Methods Classical AutoFE methods can readily be combined with our method, one simply runs CAAFE first and then lets a classical AutoFE method find further feature extensions, as we did in Table 2. For less powerful downstream classifiers, namely Logistic Regression and Random Forests, we observe that applying AutoFE additionally to CAAFE improves performance further. The AutoML method TabPFN on the other hand is not improved by applying the evaluated AutoFE methods. This discrepancy might stem from the larger hypothesis space (complexity) of TabPFN, it can get all necessary information from the data directly. For all combinations of classifiers and additional AutoFE methods, we can see that CAAFE improves performance on average.

6 Conclusion

Our study presents a novel approach to integrating domain knowledge into the AutoML process through Context-Aware Automated Feature Engineering (CAAFE). By leveraging the power of large language models, CAAFE automates feature engineering for tabular datasets, generating semantically meaningful features and explanations of their utility. Our evaluation demonstrates the effectiveness of this approach, which complements existing automated feature engineering and AutoML methods.

This work emphasizes the importance of context-aware solutions in achieving robust outcomes. We expect that LLMs will also be useful for automating other aspects of the data science pipeline, such as data collection, processing, model building, and deployment. As large language models continue to improve, it is expected that the effectiveness of CAAFE will also increase.

Dataset descriptions play a critical role in our method; however, in our study, they were derived solely from web-crawled text associated with public datasets. If users were to provide more accurate and detailed descriptions, the effectiveness of our approach could be significantly improved.

However, our current approach has some limitations. Handling datasets with a large number of features can lead to very large prompts, which can be challenging for LLMs to process effectively. The testing procedure for adding features is not based on statistical tests, and could be improved using techniques of previous feature engineering works. LLMs, at times, exhibit a phenomenon known as "hallucinations.", where models produce inaccurate or invented information. Within CAAFE, this might result in the generation of features and associated explanations that appear significant and are logically presented, even though they may not be grounded in reality. Such behavior can be problematic, especially when individuals place trust in these systems for essential decision-making or research tasks. Finally, the usage of LLMs in automated data analysis comes with a set of societal and ethical challenges. Please see Section B for a discussion on the broader impact and ethical considerations.

Future research may explore prompt tuning, fine-tuning language models, and automatically incorporating domain-knowledge into models in other ways. Also, there may lie greater value in the interaction of human users with such automated methods, also termed human-in-the-loop AutoML (Lee & Macke, 2020), where human and algorithm interact continuously. This would be particularly easy with a setup similar to CAAFE, as the input and output of the LLM are interpretable and easily modified by experts.

References

Anaconda. The state of data science 2020. Website, 2020. URL https://www.anaconda.com/ state-of-data-science-2020 .

Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand, Edward Raff, Kanika Madan, Vikram Voleti, Samira Ebrahimi Kahou, Vincent Michalski, Tal Arbel, Chris Pal, Gael Varoquaux, and Pascal Vincent. Accounting for variance in machine learning benchmarks. In A. Smola, A. Dimakis, and I. Stoica (eds.), Proceedings of Machine Learning and Systems , volume 3, pp. 747769, 2021. URL https://proceedings.mlsys.org/paper_files/paper/2021/file/ cfecdb276f634854f3ef915e2e980c31-Paper.pdf .

Leo Breiman. Random forests. Machine learning , 45:5-32, 2001.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Adam Crockett. Ai generated code creates a new security attack vector, April 2023. URL https://dev.to/adam_cyclones/ ai-generated-code-creates-a-new-security-attack-vector-39if .

Tijl De Bie, Luc De Raedt, José Hernández-Orallo, Holger H Hoos, Padhraic Smyth, and Christopher KI Williams. Automating data science. Communications of the ACM , 65(3):76-87, 2022.

Matthias Feurer, Jan N. van Rijn, Arlind Kadra, Pieter Gijsbers, Neeratyoy Mallik, Sahithya Ravi, Andreas Mueller, Joaquin Vanschoren, and Frank Hutter. Openml-python: an extensible python api for openml. arXiv , 1911.02490. URL https://arxiv.org/pdf/1911.02490.pdf .

Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. Tabllm: Few-shot classification of tabular data with large language models, 2023.

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. CoRR , abs/2103.03874, 2021. URL https://arxiv.org/abs/2103.03874 .

Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. Tabpfn: A transformer that solves small tabular classification problems in a second. arXiv preprint arXiv:2207.01848 , 2022.

Franziska Horn, Robert Pack, and Michael Rieger. The autofeat python library for automatic feature engineering and selection. CoRR , abs/1901.07329, 2019. URL http://arxiv.org/abs/1901. 07329 .

Hutter, L. Kotthoff, and J. Vanschoren (eds.). Automated Machine Learning: Methods, Systems, Challenges . Springer, 2019. Available for free at http://automl.org/book.

James Max Kanter and Kalyan Veeramachaneni. Deep feature synthesis: Towards automating data science endeavors. In 2015 IEEE international conference on data science and advanced analytics (DSAA) , pp. 1-10. IEEE, 2015.

Udayan Khurana, Deepak Turaga, Horst Samulowitz, and Srinivasan Parthasrathy. Cognito: Automated feature engineering for supervised learning. In 2016 IEEE 16th International Conference on Data Mining Workshops (ICDMW) , pp. 1304-1307, 2016. doi: 10.1109/ICDMW.2016.0190.

Udayan Khurana, Horst Samulowitz, and Deepak Turaga. Feature engineering for predictive modeling using reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Doris Jung-Lin Lee and Stephen Macke. A human-in-the-loop perspective on automl: Milestones and the road ahead. IEEE Data Engineering Bulletin , 2020.

Avanika Narayan, Ines Chami, Laurel Orr, Simran Arora, and Christopher Ré. Can foundation models wrangle your data?, 2022.

Fatemeh Nargesian, Horst Samulowitz, Udayan Khurana, Elias B Khalil, and Deepak S Turaga. Learning feature engineering for classification. In Ijcai , volume 17, pp. 2529-2535, 2017.

OpenAI. Gpt-4 technical report, 2023a.

OpenAI. openai/openai-cookbook: Examples and guides for using the openai api. https://github. com/openai/openai-cookbook , 2023b. (Accessed on 04/20/2023).

OpenAI Community. GPT-3 can't count syllables - or doesn't 'get' haiku. https://community. openai.com/t/gpt-3-cant-count-syllables-or-doesnt-get-haiku/18733 , 2021. Accessed on: 2023-03-21.

Vinay Uday Prabhu and Abeba Birhane. Large image datasets: A pyrrhic win for computer vision?, 2020.

Inioluwa Deborah Raji, Emily M. Bender, Amandalynne Paullada, Emily Denton, and Alex Hanna. Ai and the everything in the whole wide world benchmark, 2021.

Chris Rohlf. Ai code generation and cybersecurity, April 2023. URL https://www.cfr.org/ blog/ai-code-generation-and-cybersecurity .

Alaina N. Talboy and Elizabeth Fuller. Challenging the appearance of machine intelligence: Cognitive bias in llms and best practices for adoption, 2023.

Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo. Openml: Networked science in machine learning. SIGKDD Explorations , 15(2):49-60, 2013. doi: 10.1145/2641190.2641198. URL http://doi.acm.org/10.1145/2641190.2641198 .

Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luís Torgo. Openml: networked science in machine learning. CoRR , abs/1407.7722, 2014. URL http://arxiv.org/abs/1407.7722 .

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR , abs/1706.03762, 2017. URL http://arxiv.org/abs/1706.03762 .

David Vos, Till Döhmen, and Sebastian Schelter. Towards parameter-efficient automation of data wrangling tasks with prefix-tuning. In NeurIPS 2022 First Table Representation Workshop , 2022. URL https://openreview.net/forum?id=8kyYJs2YkFH .

Hilde Weerts, Florian Pfisterer, Matthias Feurer, Katharina Eggensperger, Edward Bergman, Noor Awad, Joaquin Vanschoren, Mykola Pechenizkiy, Bernd Bischl, and Frank Hutter. Can fairness be automated? guidelines and opportunities for fairness-aware automl, 2023.

Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 , 2021.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023.

Wes McKinney. Data Structures for Statistical Computing in Python. In Stéfan van der Walt and Jarrod Millman (eds.), Proceedings of the 9th Python in Science Conference , pp. 56 - 61, 2010. doi: 10.25080/Majora-92bf1922-00a.

Svante Wold, Kim Esbensen, and Paul Geladi. Principal component analysis. Chemometrics and intelligent laboratory systems , 2(1-3):37-52, 1987.

Jianyu Zhang, Jianye Hao, Françoise Fogelman-Soulié, and Zan Wang. Automatic feature engineering by deep reinforcement learning. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems , pp. 2312-2314, 2019.

Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 , 2019.

B Broader Impact Statement

B.1 Social Impact of Automation

The broader implications of our research may contribute to the automation of data science tasks, potentially displacing workers in the field. However, CAAFE crucially depends on the users inputs for feature generation and processing and provides an example of human-in-the-loop AutoML. The automation of routine tasks could free up data scientists to focus on higher-level problem-solving and decision-making activities. It is essential for stakeholders to be aware of these potential consequences, and to consider strategies for workforce education and adaptation to ensure a smooth transition as AI technologies continue to evolve.

B.2 Replication of Biases

AI algorithms have been observed to replicate and perpetuate biases observed in their training data distribution. CAAFE leverages GPT-4, which has been trained on web crawled data that contains existing social biases and generated features may be biased on these biases. An example study of such biases in web-crawled data was done by Prabhu & Birhane (2020). When data that contains demographic information or other data that can potentially be used to discriminate against groups, we advise not to use CAAFE or to proceed with great caution, double checking the generated features.

We would like to mention the best practice studies on fairness for AutoML by Weerts et al. (2023) and on bias in the use of LLMs by Talboy & Fuller (2023).

Weerts et al. (2023) give recommendations for what AutoML developers can do to improve fairnessconscious AutoML, and we follow the three most emphasized closely. i) We clearly emphasize the limitations and biases of our method. i) We emphasize the 'Usefulness' comment that our prompt asks the LLM to write into our UI to 'help users identify sources of fairness-related harm', as Weerts et al. (2023) write. ii) We execute code only after asking the user for explicit confirmation that it does not contain dangerous actions or biases; these kinds of design decisions are called "seam-full design" by (Weerts et al., 2023).

(Talboy & Fuller, 2023) give recommendations on how to best use LLMs. Their recommendation, which we as interface developers can more easily follow, is the one they emphasize the most. Namely, that 'LLMs [...] should be used as decision support tools, not final decision makers. We follow this closely by not allowing the LLM to execute code without the user's explicit consent.

The choice of benchmarks can also introduce dangerous biases in machine learning methods, as there are only a few that dominate the landscape (Raji et al., 2021). We address this problem by using datasets from two different sources (Kaggle and OpenML). In addition, we use multiple datasets from each source.

How Can Biases Arise in CAAFE? Since we are using a trained model (the LLM) to define the training setup (features used) of a downstream classifier trained on potentially biased data, we can have bias at three levels: i) at the level of the LLM, and ii) at the level of the generated features, ii) at the level of the downstream classifier.

The level of the downstream classifier is affected by the previous two levels, but is not directly affected by CAAFE, so we do not discuss it in detail. Here, as in any data science application, the user must be careful to use unbiased data. The way we set up CAAFE has some robustness against biases in the first two steps, which are controlled by CAAFE:

i) We use GPT-4, where great care has been taken to mitigate biases - still it does contain biases in its generations. We add a statement that choosing a language model that is trained to mitigate biases is a crucial step for the user to tackle this problem.

ii) Feature engineering operations by the LLM are only retained if they lead to improvements in cross-validation, due to the way our method is set up (we generate code and verify that it improves performance at each step). This gives us a slightly better defense against bias than using the LLM outputs directly, since biases that are present in the LLM but not in the data are discarded.

However, since the biases of the LLM and the dataset are most likely not independent, there is still a risk that the two will be aligned. To illustrate this risk and make users aware of potential problems, we include an illustrative Example (see end of this reply).

Simple Example We have built a fake dataset that has only one feature, a person's name, and as output we want to predict whether the person is a doctor or a nurse. The description of the dataset used in our challenge is

In all cases in this sample a name commonly associated with male gender is associated with a doctor. Additionally, we only used female names ending in an "a".

We can see that GPT-4 tends to suggest an attribute that uses the typically gender-associated ending of the name to classify, i.e. it generates the following output

We can see that our prompt actually makes it very easy to detect the bias of the LLM in this case, since the 'Usefulness' comment the LLM is asked to provide, at least in this non-cherry-picked example, just gives away its bias.

B.3 AI Model Interpretability

As the adoption of advanced AI methods grows, it becomes increasingly important to comprehend and interpret their results. Our approach aims to enhance interpretability by providing clear explanations of model outputs and generating simple code, thus making the automated feature engineering process more transparent.

B.4 Risk of increasing AI capabilities

We do not believe this research affects the general capabilities of LLMs but rather demonstrates their application. As such we estimate our work does not contribute to the risk of increasing AI capabilities.

C Reproducibility

Code release In an effort to ensure reproducibility, we release code to reproduce our experiments at https://github.com/automl/CAAFE We release a minimal demo at a simple demo.

Availability of datasets All datasets used in our experiments are freely available at OpenML.org (Vanschoren et al., 2014) or at kaggle.com , with downloading procedures included in the submission.

D Full LLM Prompt

Figure 5 shows the full prompt for one examplary dataset. The generated prompts are in our repository: https://github.com/automl/CAAFE/tree/main/data/generated_code .

E Additional Results

E.1 Semantic Blinding

Semantic information, i.e. the context of the dataset and its columns, is crucial and can only be captured through laborious human work or our novel approach of using LLMs - this is the core of our approach. To further verify and quantify this claim, we perform an experiment where the context of the dataset is left out (i.e. feature names and dataset description are not given to the LLM). We find a strong drop in performance from an average AUROC of 0.822 to 0.8 over all datasets for GPT-4.

E.2 Per Dataset Results

E.3 Generated Prompts and Code

You can find the generated prompts and the respective LLM generated code in our repository: https://github.com/cafeautomatedfeatures/CAFE/tree/main/data/generated_code .

F Compute

1

2

3

4

5

6

7

8

9

10

Number of iterations

0.8075

0.8100

0.8125

0.8150

0.8175

0.8200

0.8225

Mean ROC AUC

0.0

0.2

0.4

0.6

Token cost in $

0

1

2

3

4

5

Time in minutes

Figure F illustrates the increasing performance but also cost and time spent for more feature engineering iterations. Prediction for LLMs is done per token and so the generation of code takes dominates the 4:43 minutes evaluation time of CAAFE on average per dataset. For GPT-3.5 this time is reduce to about 1/4. Also for GPT-3.5 the cost is reduced to 1/10 as of the writing of this paper. For the evaluation of TabPFN we use one Nvidia RTX 2080 Ti as well as 8 Intel(R) Xeon(R) Gold 6242 CPU @2.80GHz CPU cores.

G Datasets

G.1 Dataset Collection and Preprocessing

OpenML datasets We use small datasets from OpenML (Vanschoren et al., 2013; Feurer et al.) that have descriptive feature names (i.e. we do not include any datasets with numbered feature names). Datasets on OpenML contain a task description that we provide as user context to our method and that we clean from redundant information for feature engineering, such as author names or release history. While some descriptions are very informative, other descriptions contain much less information. We remove datasets with more than 20 features, since the prompt length rises linearly with the number

of features and exceeds the permissible 8,192 tokens that standard GPT-4 can accept. We show all datasets we used in Table 6 in Appendix G. When datasets are perfectly solvable with TabPFN alone (i.e. reaches ROC AUC of 1.0) we reduce the training set size for that dataset to 10% or 20% of the original dataset size. This is the case for the datasets 'balance-scale' (20%), 'breast-w' (10%) and 'tic-tac-toe' (10%). We focus on small datasets with up to 2 000 samples in total, because feature engineering is most important and significant for smaller datasets.

Kaggle datasets We additionally evaluate CAAFE on 4 datasets from Kaggle that were released after the knowledge cutoff of our LLM Model. These datasets contain string features as well. String features allow for more complex feature transformations, such as separating Names into First and Last Names, which allows grouping families. We drop rows that contain missing values for our evaluations. Details of these datasets can also be found in Table 6 in Appendix G.

G.2 Dataset Descriptions

The dataset descriptions used were crawled from the respective datasource. For OpenML prompts uninformative information such as the source or reference papers were removed. Figures 20 show the parsed dataset descriptions used for each dataset.

**Eucalyptus

Soil

Conservation**

The objective was to determine which seedlots in a species are best for soil conservation in seasonally dry hill country. Determination is found by measurement of height , diameter by height , survival , and other contributing factors.

It is important to note that eucalypt trial methods changed over time; earlier trials included mostly 15 - 30cm tall seedling grown in peat plots and the later trials have included mostly three replications of eight trees grown. This change may contribute to less significant results.

Experimental data recording procedures which require noting include:

instances with no data recorded due to experimental recording procedures require that the absence of a species from one replicate at a site was treated as a missing value , but if absent from two or more replicates at a site the species was excluded from the site 's analyses.

missing data for survival , vigour , insect resistance , stem form , crown form and utility especially for the data recorded at the Morea Station; this could indicate the death of species in these areas or a lack in collection of data.

Attribute Information

Abbrev - site abbreviation - enumerated

Rep - site rep - integer

Locality - site locality in the North Island - enumerated

Map_Ref - map location in the North Island - enumerated

Latitude - latitude approximation - enumerated

Altitude - altitude approximation - integer

Rainfall - rainfall (mm pa) - integer

Frosts - frosts (deg. c) - integer

Year - year of planting - integer

Sp - species code - enumerated

PMCno - seedlot number - integer

DBH - best diameter base height (cm) - real

Ht - height (m) - real

Surv - survival - integer

Vig - vigour - real

Ins_res - insect resistance - real

Stem_Fm - stem form - real

Crown_Fm - crown form - real

Brnch_Fm - branch form - real Class:

Utility - utility rating - enumerated

Relevant papers

Bulluch B. T., (1992) Eucalyptus Species Selection for Soil Conservation in Seasonally Dry Hill Country - Twelfth Year Assessment New Zealand Journal of Forestry Science 21(1): 10 - 31 (1991)

Kirsten

Thomson

and

J.

McQueen

(1996)

Machine

Learning

Applied

to

Fourteen

Agricultural

Datasets.

University

of

Waikato

Research

Report

https://www.cs.waikato.ac.nz/ml/publications/1996/Thomson -McQueen -96.pdf + the original publication:

Binarized version of the original data set (see version 1). The multi -class target feature is converted to a two-class nominal target feature by re-labeling the majority class as positive ('P') and all others as negative ('N'). Originally converted by Quan Sun.

Dataset

Description

In

this

competition

your

task

is

to

predict

whether

a

passenger

was

transported

to

an

alternate

dimension

during

the

Spaceship

Titanic 's

collision

with

the

spacetime

anomaly.

To

help

you

make

these

predictions ,

you're

given

a

set

of

personal

records

recovered

from

the

ship 's

damaged

computer

system.

File

and

Data

Field

Descriptions

train.csv

-

Personal

records

for

about

two-thirds

(~8700)

of

the

passengers ,

to

be

used

as

training

data.

PassengerId

-

A

unique

Id

for

each

passenger.

Each

Id

takes

the

form

gggg_pp

where

gggg

indicates

a

group

the

passenger

is

travelling

with

and

pp

is

their

number

within

the

group.

People

in

a

group

are

often

family

members ,

but

not

always.

HomePlanet

-

The

planet

the

passenger

departed

from ,

typically

their

planet

of

permanent

residence.

CryoSleep

-

Indicates

whether

the

passenger

elected

to

be

put

into

suspended

animation

for

the

duration

of

the

voyage.

Passengers

in

cryosleep

are

confined

to

their

cabins

.

Cabin

-

The

cabin

number

where

the

passenger

is

staying.

Takes

the

form

deck/num/side,

where

side

can

be

either

P

for

Port

or

S

for

Starboard.

Destination

-

The

planet

the

passenger

will

be

debarking

to.

Age

-

The

age

of

the

passenger.

VIP

-

Whether

the

passenger

has

paid

for

special

VIP

service

during

the

voyage.

RoomService ,

FoodCourt ,

ShoppingMall ,

Spa,

VRDeck

-

Amount

the

passenger

has

billed

at

each

of

the

Spaceship

Titanic 's

many

luxury

amenities.

Name

-

The

first

and

last

names

of

the

passenger.

Transported

-

Whether

the

passenger

was

transported

to

another

dimension.

This

is

the

target ,

the

column

you

are

trying

to

predict.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes CAAFE, a Context aware automated feature engineering approach to support auto ML by utilizing Large Language Models (LLMs). Proposed method generates new features or transforming the feature space from tabular data by using LLMs and incorporate them to train new models. Transforming the feature space can be an iterative process where in each step LLM generates code that is run on training and evaluation sets, followed by fitting a new model and evaluating its performance. LLM receives a set of instructions through a prompt which includes semantic information about the data set, feature names, their data types, useful stats about features, and a few sample values.\nThe paper reports experimental results on two sets of data sets: 1- OpenML data sets published before Sept 2021 (GPT 3.5 and 4 training date); 2- Kaggle data sets published after Sept 2021. Results show that CAAFE can improve the performance of a state of the art classifier. They also show that CAAFE can help build models that outperform models built with the help of other state of the art automatic feature generation.\n\nSTRENGTHS:\nAlthough the seminal idea proposed in the paper is rather straightforward, it's novelty makes it a worthwhile body of research for consideration.\n\n- originality: The work is novel and original. It introduces the use of LLMs for auto ML through adding contextual information which is less explored.\n- quality: The paper is technically sound and experimental results support the claims. Limitation and risks of the work are also clearly called out.\n- clarity: The paper is easy to read and follow and the contributions are discussed clearly.\n- significance: Experimental results show LLM's usefulness for feature engineering aspect of AutoML which testifies the significance of the work. For more detailed comments on submission's significance refer to the limitation section.\n\nWEAKNESSES:\n- Alternative feature generation methods that are considered in the experiments are limited. AutoFE is the main alternative method. Other techniques such as PCA or deep sparse feature learning (authors have mentioned that with small data sets they don't expect much benefit from deep learning methods but it would be interesting to emperically validate this hypothesis for the studied data sets) is not considered.\n- The human-in-the-loop aspect of CAAFE is not fully discussed. For example, one of the implications of having human in the loop is correcting the potential mistakes that LLM may make. In the paper it is not discussed how many times the generated code by LLM needed to be revised by human, not necessarily due to run time errors but because of the semantic issues in the generated features.\n\nQUESTIONS:\n- How do you make sure the generated feature makes sense? e.g. https://github.com/cafeautomatedfeatures/CAFE/blob/main/data/generated_code/airlines_v3_0_code.txt why Euclidean distance between airports even make sense? Based on the data set description it is not clear if from or to airport attributes represent distance to a canonical point.\n- Minor, line 195, please fix the question marks in \"Table ??\".\n\nLIMITATIONS:\nThe main limitation of the work is stated by authors as \"We focus on small datasets with up to 2,000 samples in total, because feature engineering is most important and significant for smaller datasets\". As stated, the proposed approach is mainly beneficial for smaller data sets which makes the method less useful for a lot of real world and specially industrial applications.\nAlong the same lines, as clearly mentioned by author(s), another limit of the work is the possibility of exceeding the prompt maximum length.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Context-Aware Automated Feature Engineering (CAAFE) for integrating domain knowledge into the AutoML process using LLMs. CAAFE automates feature engineering for tabular datasets, generating Python code that generates semantically meaningful features based on the dataset and a textual description of the data.  The study demonstrates the effectiveness of CAAFE on a range of benchmark tasks.\n\nSTRENGTHS:\n- Well written and easy to follow. Thank you!\n- Simple and very applicable to real-world scenarios!\n\nWEAKNESSES:\n- If set up correctly, the method really can't fail. Features are only added, if they improve the objective. Thus the scientific value is somewhat limited to showing that LLMs can create features that improve performance (expected) and the extent to which (here is where I see most value). Therefore comparison to other feature engineering methods seems really important and could be more extensive.\n- Fig 2 is not clear. Is this a flow from left to right? How does the bottom part relate to the boxes? Please explain better. \n\nSuggestions:\n- I don't think the phrase \"code as interface\" is self-explanatory. Maybe can be introduced explicitly\n- In section 2.1., the sentence \"GPT-4 is a deep neural network that uses a transformer architecture\" - may not actually be correct. It is rumoured that GPT4 consists of multiple models and uses some mixture of experts methodology. In any case, this information is not published which might be worthwhile pointing out.\n- l195, table reference is missing\n\nQUESTIONS:\n- What about the dataset where performance does not improve or even weaken? Is there an explanation.\n\nLIMITATIONS:\n- The main limitation is more extensive study of competing approaches. In essence the authors work really doesn't answer any open scientific questions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a feature engineering method that builds upon LLMs. Features are engineering in an iterative process of prompting the LLM to generate code for new features, evaluating the features with a ML model and generating a new prompt. Hence, the overall approach follows the common wrapper approach. In their empirical evaluation they find feature engineering via LLMs to outperform no feature engineering.\n\nSTRENGTHS:\n- To the best of my knowledge, this is the first work to employ LLMs (GPT) for automating feature engineering.\n- The general idea seems to work out quite nicely and compare comeptitively to other feature engineering methods and preferably over engineering no features.\n- The paper is well written and easy to follow. The overall presentation is really excellent - maybe due to the use of ChatGPT for the editing?\n\nWEAKNESSES:\nAlthough the idea in general is quite interesting, I have several doubts regarding this work. The first and foremost doubt is about how to sort this work into the already existing literature and whether it really establishes a new state of the art. In particular, I am thinking about the following two works which have been missed also during the discussion of related works:\nLi, Liyao, et al. \"Learning a Data-Driven Policy Network for Pre-Training Automated Feature Engineering.\" The Eleventh International Conference on Learning Representations. 2022.\nZhang, Tianping, et al. \"OpenFE: Automated Feature Generation beyond Expert-level Performance.\" arXiv preprint arXiv:2211.12507 (2022).\n(https://openreview.net/forum?id=1H1irbEaGV)\nThis might be due to these works being rather recent. However, preprints of these papers have already been available for some time. I believe a comparison to these methods is inevitable to prove the real benefit of LLMs and whether the semantic information of feature names is really needed to come up with better features.\n\nAnother issue with the paper is that it claims that LLMs are taking advantage of the semantical information of a dataset, e.g., the names of the features etc. But this is not substantiated in the experiments nor visible in the explanations generated in the examples.\n\nMoreover, I am a little bit doubtful to what extent the explanations of an LLM can be meaningful when LLMs base the text generated on the likelihood of the next word. In particular, logical argumentation does not seem to be a strengths of GPT so far and the flaws in logical reasoning can be made easily visible. E.g. testing with a sentence like \"The [doctor/nurse] yelled at the [doctor/nurse] because she was late. Who was late?\" and asking for an explanation for the decision. Why should the explanations be more meaningful for datasets?\n\nSpeaking about the semantics that are supposedly extracted from feature names, it would have been interesting to see whether this information is indeed used, e.g., by blinding feature names. Neither in the main paper nor in the supplement I could find experiments that aim at answering these questions. However, these experiments lacking mean an unproved claim made in the paper.\n\nMinor:\n- Reference to table in line 195 broken.\n\nQUESTIONS:\n1. How does CAAFE compare to the SOTA in automated feature engineering?\n2. What makes CAAFE semi-automated? As far as I understood, CAAFE only requires an input by the user once in the beginning to give a description of the dataset. If this is the reason for semi-automation are not every AutoML tools out there semi-automated?\n3. Do LLMs in CAAFE really leverage semantic information about the dataset?\n4. As the authors already state in their Broader Impact Statement there is a certain risk that biases contained in LLMs transfer to the engineered features and that CAAFE then builds features based on these biases. Should not every feature engineering approach based on LLMs directly incorporate mechanisms to prevent to leverage such biases? Especially since such biases might be very subtle.\n\nLIMITATIONS:\nThe authors prove a broader impact statement estimating what effect and potential negative societal impact there might be.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel approach to automated feature engineering utilizing large language models (LLMs). Authors propose to use LLMs to generate code for feature generation. In the proposed method CAAFE, LLM is given a prompt with dataset description and a task of writing code for feature generation, the new feature is accepted if the validation score improves after training an ML model on a dataset containing the new feature. This procedure is repeated.\n\nThe paper demonstrates the effectiveness of the approach by testing on 10 public datasets from OpenML and 4 datasets from Kaggle (less likely to leak into the LLM training set) and slightly improving the overall classification accuracy of the strong tabular classification model TabPFN\n\nSTRENGTHS:\n- I find the idea of incorporating LLMs via code generation to tabular data automatic feature engineering clever and clean\n- Having interpretable features is a big plus and an advantage over prior automatic methods\n- If I'm not mistaken, the first positive showing of LLMs in tabular data (where using an LLM seem to improve the performance of an already strong tabular model)\n- Percussion taken to create a non-leaked test-set\n\nWEAKNESSES:\nThe major issue (that's why the score is not as high as it could have been for me) is the lacking evaluation. Both in terms of baselines/complementary methods and datasets. \n- Authors compare and complement the proposed method with AutoFeat `[1]` and DFS `[2]`, while much more effective automatic feature engineering methods like FETCH `[3]` and OpenFE `[4]` exist. \n- Also, in the above-mentioned papers, the evaluations are much more extensive, including more datasets (also regression datasets) and base models. Implementing CAAFE in those benchmarks would be a very big plus for the validation of the proposed method.\n\nQUESTIONS:\n- What do the bold entries mean in the table? The standard deviation values are pretty large, but results are bolded. Were stat tests used?\n- Does performance saturate after 10 iterations? What is the reason for stopping there?\n- Do you update the prompt with the generated feature descriptions for new iterations (I think info is missing in the paper, but this seems important)?\n- Could you report some numerical quantification of features generated by CAAFE, like feature importance ranking?\n- Minor issues:\n  - line 159: should probably cite TabPFN\n  - line 195: broken reference\n\nLIMITATIONS:\n- I think the fact that method requires dataset and feature descriptions is a limitation\n- Looking at the generated code, some features and their usefulness explanations are, almost surely, hallucinations. I think It's important to mention hallucinations in the limitations section to represent this. \n\n**References**\n- `[1]` Franziska Horn, Robert Pack, and Michael Rieger. The autofeat python library for automatic feature engineering and selection.\n- `[2]` James Max Kanter and Kalyan Veeramachaneni. Deep feature synthesis: Towards automating data\nscience endeavors\n- `[3]` Li, Liyao, et al. \"Learning a Data-Driven Policy Network for Pre-Training Automated Feature Engineering.\" \n- `[4]` Zhang, Tianping, et al. \"OpenFE: Automated Feature Generation beyond Expert-level Performance.\"", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper proposed Context-Aware Automated Feature Engineering (CAAFE) approach to integrate new features learned from dataset descriptions using large language models into AutoML process for tabular datasets. The proposed approach was evaluated using 14 datasets.

STRENGTHS:
The paper is well-written with clear research motivation, experiment design and results illustration. It's an interesting attempt to try to integrate large language models into AutoML process. 
Code scripts are provided.

WEAKNESSES:
1. Overall the use case of adding features from data description is trivial, even using large language models or human-in-the-loop concept. The proposed feature engineering process through prompt generation from the data description or any semantic information looks more of heuristic rules in featurization, which could be performed without large language models. LLM plays the role of automated code generation, instead of machine learning feature engineering approaches (e.g. PCA) incorporated in most AutoML systems. 
2. The paper fails to catch up with the latest trends of AutoML systems. Most of existing AutoML systems will perform automated feature engineering, e.g. H2O AutoML, TPOT, AutoGluon, AutoSklearn, etc. Missing those AutoML baselines significantly eliminate the value of the proposed approach in modeling effectiveness. 
3. In Table 1, the increased CAAFE performance is relatively trivial for most datasets, except the tic-tac-toe, which significantly impact the mean ROC AUC.

QUESTIONS:
Highly recommend the author to run the data experiments using the existing AutoML systems (listed in weakness part, they are all open source) without LLM featurizations from dataset description as baselines.

LIMITATIONS:
Yes. The author addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07