VLM^2-Bench: A Closer Look at How Well VLMs Implicitly Link Explicit
  Matching Visual Cues
Authors: name: Jianshu Zhang; name: Dongyu Yao; name: Renjie Pi; name: Paul Pu Liang; name: Yi R.; name: Fung
Published: 2025-02-17 17:57:50

Abstract: Visually linking matching cues is a crucial ability in daily life, such as
identifying the same person in multiple photos based on their cues, even
without knowing who they are. Despite the extensive knowledge that
vision-language models (VLMs) possess, it remains largely unexplored whether
they are capable of performing this fundamental task. To address this, we
introduce VLM^2-Bench, a benchmark designed to assess whether VLMs can
Visually Link Matching cues, with 9 subtasks and over 3,000 test cases.
Comprehensive evaluation across eight open-source VLMs and GPT-4o, along with
further analysis of various language-side and vision-side prompting methods,
leads to a total of eight key findings. We identify critical challenges in
models' ability to link visual cues, highlighting a significant performance gap
where even GPT-4o lags 34.80% behind humans. Based on these insights, we
advocate for (i) enhancing core visual capabilities to improve adaptability and
reduce reliance on prior knowledge, (ii) establishing clearer principles for
integrating language-based reasoning in vision-centric tasks to prevent
unnecessary biases, and (iii) shifting vision-text training paradigms toward
fostering models' ability to independently structure and infer relationships
among visual cues.