Automating Benchmark Design
Authors: name: Amanda Dsouza, name: Harit Vishwakarma, name: Zhengyang Qi, name: Justin Bauer, name: Derek Pham, name: Thomas Walshe, name: Armin Parchami, name: Frederic Sala, name: Paroma Varma
Organization: Snorkel AI

The rapid progress and widespread deployment of LLMs and LLM-powered agents
has outpaced our ability to evaluate them. Hand-crafted, static benchmarks are
the primary tool for assessing model capabilities, but these quickly become
saturated. In contrast, dynamic benchmarks evolve alongside the models they
evaluate, but are expensive to create and continuously update. To address these
challenges, we develop BeTaL (Benchmark Tuning with an LLM-in-the-loop), a
framework that leverages environment design principles to automate the process
of dynamic benchmark design. BeTaL works by parameterizing key design choices
in base benchmark templates and uses LLMs to reason through the resulting
parameter space to obtain target properties (such as difficulty and realism) in
a cost-efficient manner. We validate this approach on its ability to create
benchmarks with desired difficulty levels. Using BeTaL, we create two new
benchmarks and extend a popular agentic benchmark tau-bench. Extensive
evaluation on these three tasks and multiple target difficulty levels shows
that BeTaL produces benchmarks much closer to the desired difficulty, with
average deviations ranging from 5.3% to 13.2% -- a 2-4x improvement over the
baselines.