GRAB: A Challenging GRaph Analysis Benchmark for Large Multimodal Models
Authors: name: Jonathan Roberts; name: Kai Han; name: Samuel Albanie
Published: 2024-08-21 17:59:32

Abstract: Large multimodal models (LMMs) have exhibited proficiencies across many
visual tasks. Although numerous well-known benchmarks exist to evaluate model
performance, they increasingly have insufficient headroom. As such, there is a
pressing need for a new generation of benchmarks challenging enough for the
next generation of LMMs. One area that LMMs show potential is graph analysis,
specifically, the tasks an analyst might typically perform when interpreting
figures such as estimating the mean, intercepts or correlations of functions
and data series. In this work, we introduce GRAB, a graph analysis benchmark,
fit for current and future frontier LMMs. Our benchmark is entirely synthetic,
ensuring high-quality, noise-free questions. GRAB is comprised of 2170
questions, covering four tasks and 23 graph properties. We evaluate 20 LMMs on
GRAB, finding it to be a challenging benchmark, with the highest performing
model attaining a score of just 21.7%. Finally, we conduct various ablations to
investigate where the models succeed and struggle. We release GRAB to encourage
progress in this important, growing domain.