Large Language Models Orchestrating Structured Reasoning Achieve Kaggle
  Grandmaster Level
Authors: name: Antoine Grosnit, name: Alexandre Maraval, name: James Doran, name: Giuseppe Paolo, name: Albert Thomas, name: Refinath Shahul Hameed Nabeezath Beevi, name: Jonas Gonzalez, name: Khyati Khandelwal, name: Ignacio Iacobacci, name: Abdelhakim Benechehab, name: Hamza Cherkaoui, name: Youssef Attia El-Hili, name: Kun Shao, name: Jianye Hao, name: Jun Yao, name: Balazs Kegl, name: Haitham Bou-Ammar, name: Jun Wang

We introduce Agent K v1.0, an end-to-end autonomous data science agent
designed to automate, optimise, and generalise across diverse data science
tasks. Fully automated, Agent K v1.0 manages the entire data science life cycle
by learning from experience. It leverages a highly flexible structured
reasoning framework to enable it to dynamically process memory in a nested
structure, effectively learning from accumulated experience stored to handle
complex reasoning tasks. It optimises long- and short-term memory by
selectively storing and retrieving key information, guiding future decisions
based on environmental rewards. This iterative approach allows it to refine
decisions without fine-tuning or backpropagation, achieving continuous
improvement through experiential learning. We evaluate our agent's apabilities
using Kaggle competitions as a case study. Following a fully automated
protocol, Agent K v1.0 systematically addresses complex and multimodal data
science tasks, employing Bayesian optimisation for hyperparameter tuning and
feature engineering. Our new evaluation framework rigorously assesses Agent K
v1.0's end-to-end capabilities to generate and send submissions starting from a
Kaggle competition URL. Results demonstrate that Agent K v1.0 achieves a 92.5\%
success rate across tasks, spanning tabular, computer vision, NLP, and
multimodal domains. When benchmarking against 5,856 human Kaggle competitors by
calculating Elo-MMR scores for each, Agent K v1.0 ranks in the top 38\%,
demonstrating an overall skill level comparable to Expert-level users. Notably,
its Elo-MMR score falls between the first and third quartiles of scores
achieved by human Grandmasters. Furthermore, our results indicate that Agent K
v1.0 has reached a performance level equivalent to Kaggle Grandmaster, with a
record of 6 gold, 3 silver, and 7 bronze medals, as defined by Kaggle's
progression system.