LLM-FE: Automated Feature Engineering for Tabular Data with LLMs as
  Evolutionary Optimizers
Authors: name: Nikhil Abhyankar, name: Parshin Shojaee, name: Chandan K. Reddy
Organization: Virginia Polytechnic Institute and State University

Automated feature engineering plays a critical role in improving predictive
model performance for tabular learning tasks. Traditional automated feature
engineering methods are limited by their reliance on pre-defined
transformations within fixed, manually designed search spaces, often neglecting
domain knowledge. Recent advances using Large Language Models (LLMs) have
enabled the integration of domain knowledge into the feature engineering
process. However, existing LLM-based approaches use direct prompting or rely
solely on validation scores for feature selection, failing to leverage insights
from prior feature discovery experiments or establish meaningful reasoning
between feature generation and data-driven performance. To address these
challenges, we propose LLM-FE, a novel framework that combines evolutionary
search with the domain knowledge and reasoning capabilities of LLMs to
automatically discover effective features for tabular learning tasks. LLM-FE
formulates feature engineering as a program search problem, where LLMs propose
new feature transformation programs iteratively, and data-driven feedback
guides the search process. Our results demonstrate that LLM-FE consistently
outperforms state-of-the-art baselines, significantly enhancing the performance
of tabular prediction models across diverse classification and regression
benchmarks.