BoxingGym: Benchmarking Progress in Automated Experimental Design and
  Model Discovery
Authors: name: Kanishk Gandhi, name: Michael Y. Li, name: Lyle Goodyear, name: Louise Li, name: Aditi Bhaskar, name: Mohammed Zaman, name: Noah D. Goodman

Understanding the world and explaining it with scientific theories is a
central aspiration of artificial intelligence research. Proposing theories,
designing experiments to test them, and then revising them based on data are
fundamental to scientific discovery. Despite the significant promise of
LLM-based scientific agents, no benchmarks systematically test LLM's ability to
propose scientific models, collect experimental data, and revise them in light
of new data. We introduce BoxingGym, a benchmark with 10 environments for
systematically evaluating both experimental design (e.g. collecting data to
test a scientific theory) and model discovery (e.g. proposing and revising
scientific theories). To enable tractable and quantitative evaluation, we
implement each environment as a generative probabilistic model with which a
scientific agent can run interactive experiments. These probabilistic models
are drawn from various real-world scientific domains ranging from psychology to
ecology. To quantitatively evaluate a scientific agent's ability to collect
informative experimental data, we compute the expected information gain (EIG),
an information-theoretic quantity which measures how much an experiment reduces
uncertainty about the parameters of a generative model. A good scientific
theory is a concise and predictive explanation. Therefore, to quantitatively
evaluate model discovery, we ask a scientific agent to explain their model and
then assess whether this explanation enables another scientific agent to make
reliable predictions about this environment. In addition to this
explanation-based evaluation, we compute standard model evaluation metrics such
as prediction errors. We find that current LLMs, such as GPT-4o, struggle with
both experimental design and model discovery. We find that augmenting the
LLM-based agent with an explicit statistical model does not reliably improve
these results.