ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended
  Capabilities
Authors: name: Adhiraj Ghosh; name: Sebastian Dziadzio; name: Ameya Prabhu; name: Vishaal Udandarao; name: Samuel Albanie; name: Matthias Bethge
Published: 2024-12-09 18:37:14

Abstract: Traditional fixed test sets fall short in evaluating open-ended capabilities
of foundation models. To address this, we propose ONEBench(OpeN-Ended
Benchmarking), a new testing paradigm that consolidates individual evaluation
datasets into a unified, ever-expanding sample pool. ONEBench allows users to
generate custom, open-ended evaluation benchmarks from this pool, corresponding
to specific capabilities of interest. By aggregating samples across test sets,
ONEBench enables the assessment of diverse capabilities beyond those covered by
the original test sets, while mitigating overfitting and dataset bias. Most
importantly, it frames model evaluation as a collective process of selecting
and aggregating sample-level tests.
  The shift from task-specific benchmarks to ONEBench introduces two
challenges: (1)heterogeneity and (2)incompleteness. Heterogeneity refers to the
aggregation over diverse metrics, while incompleteness describes comparing
models evaluated on different data subsets. To address these challenges, we
explore algorithms to aggregate sparse measurements into reliable model scores.
Our aggregation algorithm ensures identifiability(asymptotically recovering
ground-truth scores) and rapid convergence, enabling accurate model ranking
with less data. On homogenous datasets, we show our aggregation algorithm
provides rankings that highly correlate with those produced by average scores.
We also demonstrate robustness to ~95% of measurements missing, reducing
evaluation cost by up to 20x with little-to-no change in model rankings. We
introduce ONEBench-LLM for language models and ONEBench-LMM for vision-language
models, unifying evaluations across these domains. Overall, we present a
technique for open-ended evaluation, which can aggregate over incomplete,
heterogeneous sample-level measurements to continually grow a benchmark
alongside the rapidly developing foundation models.