Evaluating Multiview Object Consistency in Humans and Image Models
Authors: name: Tyler Bonnen; name: Stephanie Fu; name: Yutong Bai; name: Thomas O'Connell; name: Yoni Friedman; name: Nancy Kanwisher; name: Joshua B. Tenenbaum; name: Alexei A. Efros
Published: 2024-09-09 17:59:13

Abstract: We introduce a benchmark to directly evaluate the alignment between human
observers and vision models on a 3D shape inference task. We leverage an
experimental design from the cognitive sciences which requires zero-shot visual
inferences about object shape: given a set of images, participants identify
which contain the same/different objects, despite considerable viewpoint
variation. We draw from a diverse range of images that include common objects
(e.g., chairs) as well as abstract shapes (i.e., procedurally generated
`nonsense' objects). After constructing over 2000 unique image sets, we
administer these tasks to human participants, collecting 35K trials of
behavioral data from over 500 participants. This includes explicit choice
behaviors as well as intermediate measures, such as reaction time and gaze
data. We then evaluate the performance of common vision models (e.g., DINOv2,
MAE, CLIP). We find that humans outperform all models by a wide margin. Using a
multi-scale evaluation approach, we identify underlying similarities and
differences between models and humans: while human-model performance is
correlated, humans allocate more time/processing on challenging trials. All
images, data, and code can be accessed via our project page.