Can Vision Language Models Infer Human Gaze Direction? A Controlled
  Study
Authors: name: Zory Zhang, name: Pinyuan Feng, name: Bingyang Wang, name: Tianwei Zhao, name: Suyang Yu, name: Qingying Gao, name: Hokin Deng, name: Ziqiao Ma, name: Yijiang Li, name: Dezhi Luo

Gaze-referential inference--the ability to infer what others are looking
at--is a critical component of a theory of mind that underpins natural human-AI
interaction. In a controlled study, we evaluated this skill across 111 Vision
Language Models (VLMs) using photos taken with manipulated difficulty and
variability, comparing performance with that of human participants (N = 65),
and analyzed behaviors using mixed-effects models. We found that 94 of the 111
VLMs failed to do better than random guessing, while humans achieved
near-ceiling accuracy. VLMs even respond with each choice almost equally
frequently. Are they randomly guessing? Although most VLMs struggle, when we
zoom in on five of the top-tier VLMs with above-chance performance, we find
that their performance declined with increasing task difficulty but varied only
slightly across different prompts and scene objects. These behavioral features
cannot be explained by considering them as random guessers. Instead, they
likely use a combination of heuristics and guessing such that their performance
is subject to the task difficulty but robust to perceptual variations. This
suggests that VLMs, lacking gaze inference capability, have yet to become
technologies that can naturally interact with humans, but the potential
remains.