BIG-Bench Extra Hard
Authors: name: Mehran Kazemi, name: Bahare Fatemi, name: Hritik Bansal, name: John Palowitch, name: Chrysovalantis Anastasiou, name: Sanket Vaibhav Mehta, name: Lalit K. Jain, name: Virginia Aglietti, name: Disha Jindal, name: Peter Chen, name: Nishanth Dikkala, name: Gladys Tyen, name: Xin Liu, name: Uri Shalit, name: Silvia Chiappa, name: Kate Olszewska, name: Yi Tay, name: Vinh Q. Tran, name: Quoc V. Le, name: Orhan Firat

Large language models (LLMs) are increasingly deployed in everyday
applications, demanding robust general reasoning capabilities and diverse
reasoning skillset. However, current LLM reasoning benchmarks predominantly
focus on mathematical and coding abilities, leaving a gap in evaluating broader
reasoning proficiencies. One particular exception is the BIG-Bench dataset,
which has served as a crucial benchmark for evaluating the general reasoning
capabilities of LLMs, thanks to its diverse set of challenging tasks that
allowed for a comprehensive assessment of general reasoning across various
skills within a unified framework. However, recent advances in LLMs have led to
saturation on BIG-Bench, and its harder version BIG-Bench Hard (BBH).
State-of-the-art models achieve near-perfect scores on many tasks in BBH, thus
diminishing its utility. To address this limitation, we introduce BIG-Bench
Extra Hard (BBEH), a new benchmark designed to push the boundaries of LLM
reasoning evaluation. BBEH replaces each task in BBH with a novel task that
probes a similar reasoning capability but exhibits significantly increased
difficulty. We evaluate various models on BBEH and observe a (harmonic) average
accuracy of 9.8\% for the best general-purpose model and 44.8\% for the best
reasoning-specialized model, indicating substantial room for improvement and
highlighting the ongoing challenge of achieving robust general reasoning in
LLMs. We release BBEH publicly at: https://github.com/google-deepmind/bbeh.