CityLens: Benchmarking Large Language-Vision Models for Urban
  Socioeconomic Sensing
Authors: name: Tianhui Liu; name: Jie Feng; name: Hetian Pang; name: Xin Zhang; name: Tianjian Ouyang; name: Zhiyuan Zhang; name: Yong Li
Published: 2025-05-31 12:25:33

Abstract: Understanding urban socioeconomic conditions through visual data is a
challenging yet essential task for sustainable urban development and policy
planning. In this work, we introduce CityLens, a comprehensive
benchmark designed to evaluate the capabilities of large language-vision models
(LLVMs) in predicting socioeconomic indicators from satellite and street view
imagery. We construct a multi-modal dataset covering a total of 17 globally
distributed cities, spanning 6 key domains: economy, education, crime,
transport, health, and environment, reflecting the multifaceted nature of urban
life. Based on this dataset, we define 11 prediction tasks and utilize three
evaluation paradigms: Direct Metric Prediction, Normalized Metric Estimation,
and Feature-Based Regression. We benchmark 17 state-of-the-art LLVMs across
these tasks. Our results reveal that while LLVMs demonstrate promising
perceptual and reasoning capabilities, they still exhibit limitations in
predicting urban socioeconomic indicators. CityLens provides a unified
framework for diagnosing these limitations and guiding future efforts in using
LLVMs to understand and predict urban socioeconomic patterns. Our codes and
datasets are open-sourced via https://github.com/tsinghua-fib-lab/CityLens.