ZeroBench: An Impossible Visual Benchmark for Contemporary Large
  Multimodal Models
Authors: name: Jonathan Roberts, name: Mohammad Reza Taesiri, name: Ansh Sharma, name: Akash Gupta, name: Samuel Roberts, name: Ioana Croitoru, name: Simion-Vlad Bogolin, name: Jialu Tang, name: Florian Langer, name: Vyas Raina, name: Vatsal Raina, name: Hanyi Xiong, name: Vishaal Udandarao, name: Jingyi Lu, name: Shiyang Chen, name: Sam Purkis, name: Tianshuo Yan, name: Wenye Lin, name: Gyungin Shin, name: Qiaochu Yang, name: Anh Totti Nguyen, name: Kai Han, name: Samuel Albanie

Large Multimodal Models (LMMs) exhibit major shortfalls when interpreting
images and, by some measures, have poorer spatial cognition than small children
or animals. Despite this, they attain high scores on many popular visual
benchmarks, with headroom rapidly eroded by an ongoing surge of model progress.
To address this, there is a pressing need for difficult benchmarks that remain
relevant for longer. We take this idea to its limit by introducing ZeroBench-a
lightweight visual reasoning benchmark that is entirely impossible for
contemporary frontier LMMs. Our benchmark consists of 100 manually curated
questions and 334 less difficult subquestions. We evaluate 20 LMMs on
ZeroBench, all of which score 0.0%, and rigorously analyse the errors. To
encourage progress in visual understanding, we publicly release ZeroBench.