Coarse Correspondence Elicit 3D Spacetime Understanding in Multimodal
  Language Model
Authors: name: Benlin Liu; name: Yuhao Dong; name: Yiqin Wang; name: Yongming Rao; name: Yansong Tang; name: Wei-Chiu Ma; name: Ranjay Krishna
Published: 2024-08-01 17:57:12

Abstract: Multimodal language models (MLLMs) are increasingly being implemented in
real-world environments, necessitating their ability to interpret 3D spaces and
comprehend temporal dynamics. Despite their potential, current top models
within our community still fall short in adequately understanding spatial and
temporal dimensions. We introduce Coarse Correspondence, a simple,
training-free, effective, and general-purpose visual prompting method to elicit
3D and temporal understanding in multimodal LLMs. Our method uses a lightweight
tracking model to find object correspondences between frames in a video or
between sets of image viewpoints. It selects the most frequent object instances
and visualizes them with markers with unique IDs in the image. With this simple
approach, we achieve state-of-the-art results on 3D understanding benchmarks
including ScanQA (+20.5\%) and a subset of OpenEQA (+9.7\%), and on long-form
video benchmarks such as EgoSchema (+6.0\%). We also curate a small diagnostic
dataset to evaluate whether MLLMs can reason about space from a described
viewpoint other than the camera viewpoint. Again, Coarse Correspondence
improves spatial perspective-taking abilities but we highlight that MLLMs
struggle with this task. Together, we demonstrate that our simple prompting
method can significantly aid downstream tasks that require 3D or temporal
reasoning.