Can Models Help Us Create Better Models? Evaluating LLMs as Data
  Scientists
Authors: name: Michał Pietruszka, name: Łukasz Borchmann, name: Aleksander Jędrosz, name: Paweł Morawiecki

We present a benchmark for large language models designed to tackle one of
the most knowledge-intensive tasks in data science: writing feature engineering
code, which requires domain knowledge in addition to a deep understanding of
the underlying problem and data structure. The model is provided with a dataset
description in a prompt and asked to generate code transforming it. The
evaluation score is derived from the improvement achieved by an XGBoost model
fit on the modified dataset compared to the original data. By an extensive
evaluation of state-of-the-art models and comparison to well-established
benchmarks, we demonstrate that the FeatEng of our proposal can cheaply and
efficiently assess the broad capabilities of LLMs, in contrast to the existing
methods.