Vision language models are blind
Authors: name: Pooyan Rahmanzadehgervi; name: Logan Bolton; name: Mohammad Reza Taesiri; name: Anh Totti Nguyen
Published: 2024-07-09 06:20:17

Abstract: Large language models with vision capabilities (VLMs), e.g., GPT-4o and
Gemini 1.5 Pro are powering countless image-text applications and scoring high
on many vision-understanding benchmarks. Yet, we find that VLMs fail on 7
visual tasks absurdly easy to humans such as identifying (a) whether two
circles overlap; (b) whether two lines intersect; (c) which letter is being
circled in a word; and (d) counting the number of circles in a Olympic-like
logo. The shockingly poor performance of four state-of-the-art VLMs suggests
their vision is, at best, like of a person with myopia seeing fine details as
blurry, and at worst, like an intelligent person that is blind making educated
guesses. Code is available at: https://vlmsareblind.github.io/