UrBench: A Comprehensive Benchmark for Evaluating Large Multimodal
  Models in Multi-View Urban Scenarios
Authors: name: Baichuan Zhou; name: Haote Yang; name: Dairong Chen; name: Junyan Ye; name: Tianyi Bai; name: Jinhua Yu; name: Songyang Zhang; name: Dahua Lin; name: Conghui He; name: Weijia Li
Published: 2024-08-30 13:13:35

Abstract: Recent evaluations of Large Multimodal Models (LMMs) have explored their
capabilities in various domains, with only few benchmarks specifically focusing
on urban environments. Moreover, existing urban benchmarks have been limited to
evaluating LMMs with basic region-level urban tasks under singular views,
leading to incomplete evaluations of LMMs' abilities in urban environments. To
address these issues, we present UrBench, a comprehensive benchmark designed
for evaluating LMMs in complex multi-view urban scenarios. UrBench contains
11.6K meticulously curated questions at both region-level and role-level that
cover 4 task dimensions: Geo-Localization, Scene Reasoning, Scene
Understanding, and Object Understanding, totaling 14 task types. In
constructing UrBench, we utilize data from existing datasets and additionally
collect data from 11 cities, creating new annotations using a cross-view
detection-matching method. With these images and annotations, we then integrate
LMM-based, rule-based, and human-based methods to construct large-scale
high-quality questions. Our evaluations on 21 LMMs show that current LMMs
struggle in the urban environments in several aspects. Even the best performing
GPT-4o lags behind humans in most tasks, ranging from simple tasks such as
counting to complex tasks such as orientation, localization and object
attribute recognition, with an average performance gap of 17.4%. Our benchmark
also reveals that LMMs exhibit inconsistent behaviors with different urban
views, especially with respect to understanding cross-view relations. UrBench
datasets and benchmark results will be publicly available at
https://opendatalab.github.io/UrBench/.