CityLens: Evaluating Large Vision-Language Models for Urban Socioeconomic Sensing
Authors: Tianhui Liu
Hetian Pang
Xin Zhang
Tianjian Ouyang
Zhiyuan Zhang
Jie Feng
Yong Li
Pan Hui
Venue: ICLR 2026 Poster (2026)
Keywords: Multi-modal Large Language Model
Socioeconomic Prediction
Urban Imagery
Urban Science
Benchmark
Primary area: datasets and benchmarks
TL;DR: We propose a global scale benchmark to evaluate the performance of large language-vision models for urban imagery-based socioeconomic prediction

Abstract: Understanding urban socioeconomic conditions through visual data is a challenging yet essential task for sustainable urban development and policy planning. In this work, we introduce CityLens, a comprehensive benchmark designed to evaluate the capabilities of Large Vision-Language Models (LVLMs) in predicting socioeconomic indicators from satellite and street view imagery. We construct a multi-modal dataset covering a total of 17 globally distributed cities, spanning 6 key domains: economy, education, crime, transport, health, and environment, reflecting the multifaceted nature of urban life. Based on this dataset, we define 11 prediction tasks and utilize 3 evaluation paradigms: Direct Metric Prediction, Normalized Metric Estimation, and Feature-Based Regression. We benchmark 17 state-of-the-art LVLMs across these tasks. These make CityLens the most extensive socioeconomic benchmark to date in terms of geographic coverage, indicator diversity, and model scale. Our results reveal that while LVLMs demonstrate promising perceptual and reasoning capabilities, they still exhibit limitations in predicting urban socioeconomic indicators. CityLens provides a unified framework for diagnosing these limitations and guiding future efforts in using LVLMs to understand and predict urban socioeconomic patterns.

Published as a conference paper at ICLR 2026
CITYLENS: EVALUATING LARGE MODELS FOR URBAN SOCIOECONOMIC SENSING
Tianhui Liu'
Hetian Pang2
Xin Zhang²
Tianjian Ouyang²
Zhiyuan Zhang"
Jie Feng 2,41 Yong Lift
Pan Hui
Information Hub, The Hong Kong University of Science and Technology (Guangzhou)
2 DDeparteent of Electronic Engineering. BNRist, Tsinghua University
School of Electronic and Information Engineering, Beijing Jiaotong University 'Zhongguaneun Academy
ABSTRACT
Understanding urban socioeconomic conditions through visual data is a challeng- ing yet essential task for sustainable urban development and policy planning. In this work, we introduce CityLens, comprehensive benchmark designed to eval- uate the capabilities of Large Vision-Language Models (LVLMs) in predicting socioeconomic indicators from satellite and street view imagery. We construct multi-modal dataset covering total of 17 globally distributed cities, spanning 6 key domains: economy, education, crime, transport, health, and environment, reflecting the multifaceted nature of urban life. Based this dataset, we define 11 prediction tasks and utilize 3 evaluation paradigms: Direct Metric Prediction, Normalized Metric Estimation, and Feature-Based Regression. We benchmark 17 state-of-the therarr LVLMs across these tasks. These make CityLens the most exten- sive socioeconomic benchmark date in terms of geographic coverage, indicator diversity, and model scale. Our results reveal that while LVLMs demonstrate promising perceptual and reasoning capabilities, they still exhibit limitations in predicting urban socioeconomic indicators. CityLens provides a unified frame- work for diagnosing these limitations and guiding future efforts in using LVL.Ms to understand and predict urban socioeconomic patterns. The code and data are available https:: //tp::/ggtttttbtt https//.coootttiiuuieaeae fib-lab/ lalbaa//CtyLenss
INTRODUCTION
Understanding the socioeconomic characteristics of urban regions is fundamental the planning, management, and sustainability of cities. Urban socioeconomic sensing, the process of quantifying indicators such as income, education, health, and transport conditions across spatial umits, plays a critical role in shaping how cities function and evolve. These indicators directly influence residents' quality of life and are deeply intertwined with key aspects of urban inequality. mobility, and resource allocation. Moreover, urban socioeconomic data serves cornerstone for measuring progress toward several United Nations Sustainable Development Goals (UN SDGs) [Nations] 2013). Accurate and timely information urban disparities essential for tracking these goals and designing effective interventions. In practice, governments and urban planners rely on socioeconomic indicators to inform wide range of decisions from zoning regulations and infrastructure investment to public health strategies. A better understanding of spatially resolved urban indicators empowers decision- makers to allocate resources more equitably, respond to local needs, and promote inclusive urban development.
growing body of work has explored the use of classical deep learning methods to predict urban sociocconomic indicators. Some approaches, such as Zhou et al. (2023), leverage knowledge graphs to infer socioeconomic indicators. In parallel, researchers have explored the use of urban imagery understand cities through their visual appearance. Methods such Li et al.) (2022) Liu al. (2023b), Lin et al. (2024), and Yong & Zhou (2024) employ contrastive learning to generate visual representations from street view or satellite images, while others apply basic computer vision models
Corresponning author, email: fengji 2ee hommmill.ommm liyong07 @wwiingus..eduue... edu.en, panhui @ust.hk.

---

Published as a conference paper at ICLR 2026
view Images Satellite Image
Suppose you are professional sodioeconomic data analyst in (olty).....
LityLens
11 Indicators
Models
Socioeconomic Prediction
3 Methods
LVLM
Economy
Transport
Crime
House Price
GDP
Public Transport
Violent Crime
Please estimate the the ceneus track.
Q: What is the average GDP for this satellite area?
the percentage of population commute by public transi??
Please estimate the occurrence violent cnime the census tract.
Drive Ratio
Population
the percentage of population commute by
Q: Please estimate "the total population satelite
driving alone?
Health
Environment
Education
Mental Health
Accessibility Health
Bachelor
Estimate the average walking-only travel time to healthcare' for this area.
Building Height
Q: Estimate the proportion of residents bathelor's degree tract
prevalence mentaal
Q: Please estimate buikding height in meters
health not good for days among adults' in census
the
Life Expectancy
Q: What's the male life expeciancy the MSOA
Figure 1: CityLenn comprises 11 socioeconomic indicators spanning six key urban domains.
to extract visual features (Fan et al. 2023 Zhang et al. 2024). However, the classical methods face several key limitations, including difficulty handling unstructured or multi-modal data, the inability to work across multiple countries, and cannot interpret subjective and culturally significant aspects of place. Nevertheless, large vision-language models are inherently equipped to address these challenges with their ability to integrate multiple modalities, generalize globally, and interpret cultural nuances.
In recent years, researchers have begun to leverage LVLMs and large language models (LLMs) to address some of the limitations of classical approaches (Hou et al. 2025 Yan et al. 2024 Hao et al.. 2025 Manvi al.) 2024bla). For example, Yan et al. (2024) and Hao et al. (2025) employ L.V.M to generate textual descriptions from urban imagery, effectively introducing textual modality to enrich visual understanding. Other studies, such Manvi al. (2024b (a). explore the ability of LLMs to predict socioeconomic indicators directly through textual prompts, and further examine issues like geographical bias across different countries. Despite these promising advances, existing works still fall short in several key aspects. Most efforts are limited in terms of spatial coverage, indicator diversity, and multi-modal integration. Crucially, there remains a lack systematic and unified benchmark to comprehensively evaluate how LVLMs perform across tasks, regions, and modalities in the context of urban socioeconomic sensing. To address these limitations, we propose CityLens, comprehensive benchmark designed to evaluate the ability of large vision-language models to predict urban socioeconomic indicators using both street view and satellite imagery. Citylens spans a total of 17 cities across multiple continents, covering 11 indicators across 6 socioeconomic domains, including economy, health, education, environment, transport, and crime. By integrating diverse data modalities and global geographic coverage, CityLens enables systematie, cross-task, and cross- region evaluation of LVLMs' capabilities in urban perception, geo-visual reasoning, and numerical estimation. Overall, our contributions are summarized as follows:
To the best of our knowledge, CityLens is the largest benchmark urban socioeconomic sensing, along geographic coverage. indicator diversity, and model scale. It covers 17 cities across different continents, indicators in 6 socioeconomic domains, using both street view and satellite imagery.
We conduct comprehensive evaluation of 17 state-of-the-art large vision-language models across diverse tasks and evaluation settings, systematically comparing three paradigms: Direct Metric Prediction, Normalized Estimation, and Feature- -Based Regression.
We design extensive experiments and provide detailed analysis that offers new insights into how input configuration, model architecture, and task design affect model performance, highlighting challenges, opportunities, and future directions for socioeconomic sensing with LVLMs.
2

---

Published as a conference paper at ICLR 2026
2 METHODS
In this paper, we present CityLens, comprehensive benchmark designed to evaluate the capabilities of large vision-language models in predicting socioeconomic indicators from both satellite and street view imagery. As illustrated in Figure [T CityLens spans 11 real-world indicators across socioeconomic domains, covering 17 globally distributed cities with diverse urban forms and development levels. To systematically assess model performance, we evaluate 17 different LVLMs using 3 distinct evaluation paradigms.
2.1 DATASET CONSTRUCTION
To support the evaluation of LVLMs across diverse socioeconomic indicators, as Figure 2 illustrates, we construct a region-level dataset by performing data collection, indicator selection, and data mapping. Each region is represented by 1 satellite image and 10 street view images, and is associated with corresponding socioeconomic indicator values.
Data Collection We provide a detailed list of data sources for all indicators in the Appendix A..5. here, we briefly describe the collected indicators. Under the economy domain, we cover critical indicators: Gross Domestic Product (GDP), house price, population, median household income, poverty 100%, poverty 200%, and income Gini coefficient. In the transport domain, we include seven indicators: PMT, VMT, PTRP, VTRP, walk and bike ratio, public transport ratio, and drive ratio. In the crime domain, we focus on two indicator: violent crime incidence and non-violent crime incidence, both defined the number of crime occurrences per census tract. For the health domain, we include 9 kinds of indicators to capture different dimensions of urban health outcomes: obesity, diabetes, cancer, no leisre-time physical activity(LPAA,, mental health, physical health, depression rate, accessibility to healthcare, and life expectancy. In the environment domain, we consider two indicators: carbon emissions and building height. Building height is increasingly used as explicit yet indirect indicator of urban socioeconomic development, population density, and land use intensity, Under the education domain, following Liu et al. (003b)), use the bachelor ratio, defined as the proportion of residents holding a bachelor's 's degree or higher, as the target variable. These domains are selected ensure balanced and holistic representation of urban conditions that are commonly studied in social science and urban planning.
Since many ground-trui indicators are only available for specific countries (the US and the UK), we focus on region-level prediction tasks in three representative cities from each country. We choose New York, San Francisco, and Chicago in the US, and Leeds, Liverpool, and Birmingham in the UK. For globally available indicators, we expand coverage to cities across 6 continents, including Cape Town, Nairobi, London, Paris, Beijing. Shanghai, Moscow, Mumbai, Tokyo, Sao Paulo, and Sydney, which ensures cross-regional evaluation diversity. Beyond ground-truth indicator data, we collect both satellite images and street view images for each task region. We obtain street view images for Beijing and Shanghai using the Baidu Maps API, while for other cities, we utilize the Google Street View API. All experimental results reported in the main paper are based on these Google and Baidu sourced street view images. To promote transparency, completeness, and reproducibility of CityLens, we further construct an alternative version of the dataset using publicly accessible street view images from Mapillary, referred to based on Mapillary street view images in Appendix A.3] Additionally, the 256 256-2266pp1 256-pixe satellite images with about 4.7 m-resolution are downloaded from Esri World Imagery.
We report the benchmark results
Indicator Selection We initially collect ground-trute -truh data for 28 indicators spanning domains. From these, as Figure 3a illustrates, we select II final indicators to construct prediction tasks. The selection followed two principles: First, we assess the perceptual relevance of indicators- i.e., whether a human could reasonably infer the variable from satellite and street view imagery. Indicators such as "Estimated personal miles traveled on a working weekday", which lack visible spatial cues, are excluded. Second, we conduct Pearson correlation analysis among semantically similar indicators in the same domain to remove redundancy. For example, in the health domain, we found high correlation between obesity and mental health (Pearson's r 0.7524), which is intuitively understandable that people experiencing psychological stress or poor mental well-being tend to overeat or engage in unhealthy eating behaviors. To avoid task redundancy, we retained only mental health in the final task list.
Data Mapping In CityLens, each region serves as a prediction unit, represented by 1 satellite image and 10 street view images, and is paired with a set of scalar labels corresponding to multiple target indicators. These labels are computed by mapping and aggregating raw tabular data from
3

---

Published as a conference paper at ICLR 2026
Data Collection
Indicator Selection
Data Mapping
Evaluation Methods
Indicator Souce
Human Test
Please astimatt the Indicator value.
CDC
Normalized
Tract
MSDA
Whether reasonable
Please eslimate the indicalor (range from 0.0 8.9).
Urban Imagery
Pearson Correlation
Global
China
Google
Street View
Baidu
Indicator A
Satalitte
Feature-Based Regression
Mapillary
Visual Input
Please provide a score for the feeluree.
Indicator B
Satellite
High
LASBO
Figure 2: Benchmark Construction Pipeline, including data collection, indicator selection, data mapping and evaluation methods.
1000
GDP
fibo
Satellite Street View
Pepolato
Task
Images
Images
Scale
Region
Envirommen
Economy
GDP
4285
42842
Global
Sat
House
Population
4517
45157
Global
Sat
Prise
Vinleni Crime
Crime
House Price
769
7770
US. UK. China CT. MSOA, Sar
LityLLnn
Public
Transport
Tiomsport
Drive Ratio
Education
Drive
Mental Health
632
6400
US
CT
Blucholer
Heabt
Ratio
Accessibility to Health
4285
42837
Global
Sat
Rato
Huntth
Monull
Life Expectancy
193
1930
UK
MSOA
Alvese
Heain
Lic
630
Bachelor Ratio
1135
11438
CT
Violent Crime
389
3960
CT
14300
Building Height
(a)
(b)
Figure 3: (a) 11 indicators benchmark and their counts. (b) Statistics of dataset.
heterogeneous sources the respective region. As shown in the region column of Figure 3b define census tract-level prediction tasks for US-only indicators, and construct MSOA-level tasks for UK-only indicators using similar strategy. For global tasks, each satellite image coverage constitutes an evaluation unit. We first download multiple satellite images covering each city's spatial extent. Then, within each satellite image's 's coverage, randomly sample 30 geographic points and collect at least 10 street view images corresponding to those points. For the China house price task, we follow the global task methodology and select Beijing and Shanghai as target cities. A detailed explanation of the mapping and aggregating of data from various sources and geographic scales for each task provided Appendix A.5.2. Due resource constraints, we randomly sample up to 500 cases per task for country-specinns indicators, and up to 1000 cases task for global indicators. In practice, some tasks contain fewer samples due to data availability limitations, but these values represent the maximum sample size allowed per task. The detailed statistics of available data before applying the sampling strategy are presented in Figure
2.2 EVALUATION METHOODOLOGIES
We design three distinct paradigms to explore the capabilities of LVLMs in socioeconomic indicator prediction. As shown in Figure 4 each paradigm is aimed evaluating different facet of how LVLMs can be applied to this task.
Direct Metric Prediction Direct Metrie Prediction refers providing region-level urban imagery and directly querying the LVLM for the metric value, such as: "What's the percentage of the population commuting by public transit in this census tract?". In addition, the prompt positions the
4

---

Published as a conference paper at ICLR 2026
Direct Metric Prediction
Satellicc Sirect View Inages in San Francisco
Feature-Based Regression
Q: Supposs you ane demog rephic analyst specializing in eduesiion alaiaiics San Freniicccoo Lidted Based on provided sutilitee imagery saversl streal pholos, please esdmate bachelor
Q: Analyze the provided street view mage For each the follwing 13 indicators, provtte scome from 0. prominance image. The murput ahould only be indicator name followed by is soore, one Indicator
representing ihs presence
census where images aey
property features,
neighbortane single espressed
Indicators:
range approséiata
neeced: Example anewen
Heavy Vehicle
Vehide
Normalized Metric Estimation
Window Opaning
Suppose demographic
specia zing
Road
statesies SanFrancisca United Stales. Based provided imagery
3 Bidowalk
Steet Fumiture
Street View Chicago, US
estimate bachelor
tract
the
Crearary- Time
lakan. Cansider faciors sush au location visbie proparty features. es. neighborhood conditon, and any other relevant details.
Crearery- - Crass Surdud
Buy
Please provite single specific number for bachelor ratio level jon scale from 0,0 to 9.9). No explanation needed. Example erswar: 8.8
Nature
Exemple:
Person 2.5
Figure 4: Prompt examples for three evaluation methodologies.
model an urban socioeconomic scientist in specific city. Despite this, the model faces significant challenges in accurately predicting the exact true values these indicators.
Normalized Metric Estimation Given the difficulty of directly predicting precise indicator values, we adopt a Normalized Metrie Estimation approach inspired by GeoLLM (Manvi al. 2024b). Specifically, we transform all indicator values into a normalized range from 0,0 to 9.9, discretized to one decimal place. The model is then prompted estimate this normalized value based on the input images. This formulation aims to investigate whether the LVLM possesses coarse- coarse-grained spatial knowledge and the ability to associate visual cues with relative indicator levels.
Feature-Based Regression In the Feature-Based Regression approach, we first design a structured prompt that guides the LVLM to evaluate each street view image along 13 predefined visual attributes, following the visual taxonomy proposed by Fan et al.] (2023). These features capture key elements of the urban environment, such as greenery, vehicle, facade, and sidewalk. For each region, represent its visual environment using 10 sampled street view images. For each visual feature, we compute the average score across these images, resulting in single feature vector that characterizes the region. These aggregated visual features are then used as inputs to a LASSO regression model, which is trained predict the ground-truth indicator values using 5-fold cross-validation setup.
3 EXPERIMENTS
3.1 OVERALL PERFORMANCE ON FEATURE-BASED REGRESSION
As shown in Table T the overall performance suggests that our benchmark poses a significant challenge for current large vision-language models. In particular, tasks such Mental Health and Bachelor Ratio exhibit low R2 scores, in some cases even approaching zero, e.g., 0.001. This highlights the difficulty of CityLenn in Feature-Base Regression method: even when leveraging visual features extracted by advanced LVLMs, the resulting representations often fail to capture the complex patterns required for accurate prediction of socioeconomic indicators. We also observe that general LVLMss underperform compared to the domain-specific contrastive learning model Urban VLP on several tasks. Therefore, improving LVLM performance for urban socioeconomic sensing remains an important and open challenge.
Challenge of the Benchmark for LVLMs
Performance Differences Across Models We observe substantial performance differences across LVLMs, reflecting how model scale, architecture, and training design influence their ability to extract meaningful visual features for downstream prediction. Comparing models within the same series but at different scales, we find that increasing model size does not always guarantee better performance. For example, Gemma3-12B achieves the best score on GDP and Life Expectancy, yet the 27B variant performs worse in these two tasks, with relative drops of 4.3% and 6.8% respectively. This counterintuitive result may be attributed to the unique nature of socioeconomic sensing tasks, which requires the model to consistently extract and score predefined set of nuanced visual features from urban imagery. When comparing models from different series with similar parameter scales, clear differences emerge. For instance, Gemma3-4B significantly outperforms Qwen2.5VL-33B in nearly all
5

---

Published as a conference paper at ICLR 2026
Table 1: Main results on the Feature-Based Regression method. The values in the table represent R² scores. "Mean" denotes the average performance across tasks, and "SD" refers to the standard deviation. In each row, bold indicates the best result, and underline denotes the second-best.
Domain Tasks
Econ.
Crime VC
Trans.
Env.
Health AH
Edu. BR
Overall
GDP
Pop.
HP
PT
DR
BH
MH
LE
Mean
SD
Baselines
UrbanCLIP
0.450
0.030
0,316
0.033
0.128
0.123
0.612
0.021
0.191
0.024
0.094
0.184
0.196
Urb
0.717
0.132
0.589
0.149
0.551
0.446
0.807
0.403
0.382
0.025
0.422
0.417
0.243
L.LLMs
Gemma3-4B
0.479
0.252
0.036
0.103
0.486
0.365
0.585
0.183
0.294
0.
0.290
0.293
0.165
Gemma3-128
0.484
0,280
0.136
0.063
0.448
0.588
0.159
0.266
0.202
0.311
0.166
Gemma3-27B
0.463
0.324
0.141
0.077
0.567
0.525
0.590
0.211
0.283
0.245
0.297
0.338
0.166
Qwen2.5VL-3B
0.372
0.157
0.169
0.029
0,362
0.262
0.513
0.172
0.247
0.006
0.001
0.210
0.158
Qwen2.5VL-7B
0.468
0.304
0,104
0.053
0,483
0.308
0.536
0,166
0.261
0.119
0.195
0.272
0.157
Qwen2.5VL-32B
0.517
0.347
0.067
0.067
0,508
0.427
0.528
0.178
0,261
0.193
0.311
0.309
0.164
Llama4-Scout
0.460
0.264
0.164
0.090
0.508
0.479
0.524
0.168
0.280
0.155
0.197
0.299
0.155
Llama4-Maverick
0.452
0.308
0.233
0.110
0,547
0.447
0.523
0.229
0,293
0.172
0.249
0.324
0.139
Mistral-small-311249
0.452
0.366
0.144
0.062
0,499
0.393
0.571
0.159
0.260
0.098
0.198
0.291
0.166
Phi-4-mulimodal
0.190
0.079
0.154
0.038
0,238
0.224
0.142
0.096
0.172
0.144
0.103
0.143
0.059
Nova-lite-vl
0.466
0.219
0,216
0.007
0.439
0.359
0.538
0.222
0.272
0,145
0.175
0.278
0.150
Minimax-01
0.447
0,336
0.197
0.068
0,523
0.448
0.516
0.113
0.273
0,162
0.170
0.295
0,159
Gemini-2.0-Flash
0.436
0.317
0.129
0.090
0,560
0.490
0.559
0.222
0.310
0.194
0.201
0.319
0.161
Gemini-2.5-Flash
0.375
0,314
0.143
0.064
0,527
0.500
0.568
0.251
0.277
0,210
0.203
0.312
0,156
GPT-4o-mini
0.425
0,251
0.119
0.076
0.470
0.253
0.554
0.239
0.295
0.236
0.163
0,280
0.141
GPT-4.1-mini
0.441
0,316
0.243
0.063
0.542
0.444
0.505
0.151
0.264
0.150
0.195
0,301
0.153
GPT-4.1-nano
0.360
0.314
0.201
0.084
0.360
0.198
0.485
0.175
0.267
0.086
0.227
0.251
0.117
tasks, with relative improvements ranging from 6.4% to 255% across different indicators, suggesting that Gemma's 's architecture or training process may enable more consistent and informative scoring of urban visual features, which in turn leads to better performance in sociocconomic prediction.
Variations Across Different Task Types Performance also varies across task types. Tasks like Building Height, Public Transport, and GDP tend to have relatively higher values across models, with Building Height reaching an R² of 0.590, suggesting that these indicators are associated with more observable visual cues that can be directly captured from street view images. For instance, Building Height closely linked to the skyline and the vertical structure visible in images; Public Transport usage may be inferred from the presence of bus stops, transit signs, or road markings. In contrast, tasks such as Life Expectancy and Mental Health remain highly challenging, exhibiting low or near-zero predictive scores for many models. These indicators are influenced by latent factors such as lifestyle, stress levels, or social cohesion, which lack clear visual signals in urban imagery. Even if certain proxies exist, such as the presence of graffiti or the amount of green space, they are often subtle or semantically ambiguous, making it hard for LVLMs to interpret reliably and consistently.
3.2 EVALUATION OF DIRECT AND NORMALIZED ESTIMATION
Overall Performance We evaluate the performance of large vision-language models on all 11 tasks using both the Direct Metric Prediction and Normalized Estimation settings. To ensure meaningful analysis, exclude model-task pairs with R < -0.5 under either setting and choose to abstract away the model identity. The final comparison is visualized in Figure where each point represents the performance of a specific model on a specific task, evaluated under both estimation settings. A few tasks such as House Price, Public Transport, and Building Height achieve relatively better πt scores under certain models and settings, e.g., House Price consistently exceeds 0.2 under the Direct setting. These tasks are likely more visually grounded, with cues such as building density, road layout, and commercial signage that can be directly observed from urban imagery. This suggests that some socioeconomic indicators may be approximated more easily when the visual-structural link is strong. However, the majority of results fall into the low or even negative R22 range, indicating that the model's predictions often fail to explain the variance in the ground-truth indicator values. This suggests that, the models may still lack the necessary numerical grounding, contextual interpretation, and semantic alignment required to associate urban visual content with structured socioeconomic quantities. Even with normalization, which alleviates the demand for precision by coarsening the prediction space, performance remains weak across most tasks. In many cases, the model predictions tend to collapse

---

Published as a conference paper at ICLR 2026
toward city-wide averages exhibit narrow output range, suggesting lack of sensitivity to fine- grained regional variation. This behavior indicates that the models may struggle to differentiate subtle socio-spatial differences across urban regions, especially when visual cues are weak or ambiguous. These findings highlight the inherent difficulty of CityLeens models struggle not only when asked pre- dict exact values, but also when the task is simplified into normalized estimation. We also incorporate the Feature- Beature-Based Regression method into a unified eval- uation. A comprehensive comparison across all three paradigms is visualized in Appendix Figure 12
1.0
GDP
Public Tuusport
Husac Price
Mestal Heall
Pepuiation
Life Expezurcy
11.-
Baclickr Raio
Acccis Hoalluase
Violent
Building Height
Drive Ratio
Normulized R*
Task Preference Between Direct and Normalized Estimation In Figure 5 the diagonal line indicates equal performance under both methods; points above it suggest that the task benefits more from normaliza- tion, while points below indicate a preference for direct estimation. This result highlights that different tasks tend to favor different estimation strategies, depend- ing the nature of the indicator and its visual and semantic properties. Specifically, tasks such as Vio- lent Crime, GDP, and Population more frequently observed above the diagonal, suggesting that these indi- cators with limited direct visual correspondence benefit from normalized formulation that emphasizes rela- tive ranking rather than precise value prediction. These tasks are difficult to estimate accurately, but models may still capture coarse ordinal relationships across regions, aided by their global knowledge priors and implicit ranking sense. Conversely, tasks like Bachelor Ratio, House Price, Public Transport, and Accessibility to Health tend to fall below the diagonal, indicating better performance under the direct estimation setting. These tasks are often associated with clearer, more stable visual correlates, such as building types, infrastructure visibility, and environmental layout, which can support more precise image-to- tore-ttt-veeee value mappings. In addition, some indicators, e.g., Life Expectancy, exhibit narrower value ranges or lower variance, making them more amenable to direct value prediction. Moreover, for tasks like House Price and Bachelor Ratio, LVLMs may leverage latent knowledge about typical value scales across different cities, enabling surprisingly accurate numerical predictions. Taken together, these findings emphasize the importance of task-specific method selection in socioeconomic indicator prediction. The CityLens benchmark thus not only tests model capacity, but also reveals the nuanced interplay between task semanties and prediction strategy.
+
0.2
0.0-
-0.3
-0.4
-0.2
0.0
0.2
0.4
0.6
1.0
Direct
Figure 5: Comparison of task-wise n2² per- formance between Direct Metric Prediction and Normalized Estimation across 11 so- cioeconomic indicators CityLens.
3.3 INFLUENCE OF GEOGRAPHIC VARIATION AND INPUT COMPOSITION
City-Level Performance Variations To better understand the variation in socioeconomic prediction outcomes across different cities, we conduct a level analysis for the GDP task under the Feature- Based Regression paradigm. Each of the 13 cities is represented by 100 regions, with Gemma3- extracting 13 visual features per street view image. Among the 13 citics evaluated in the GDP prediction task, we observe considerable variation in model performance in Figure [aa Cities such as Shanghai, San Francisco, and Sao Paulo achieve R scores above 0.43, indicating relatively strong predictive performance. One possible explanation for the strong performance in citics like Shanghai lies in their well-structured urban design and high alignment between street-level appearance and economic development. These cities tend to have clear visual stratification between affluent and less affluent areas, consistent architectural patterns and homogeneous zoning that make features more learnable and high quality, diverse street view coverage. In contrast, cities like Mumbai and Moscow yield near-zero even negative R², which may be attributed two key factors. First, there may be weak alignment between street-level visuals and actual economic activity, especially in cities with spatially mixed development, where wealth and poverty coexist within the same region, blurring the visual economic signal. Second, the quality and coverage of street view images can be a limiting factor. Inconsistent image sources, low resolution, or sparse sampling reduce the availability of reliable visual cues, hindering feature extraction and degrading downstream prediction.
Impact of Input Modalities In this part, we evaluate the impact of input modalities by comparing model performance in three configurations: using both, only street view, and only satellite imagery.
7

---

Published as a conference paper at ICLR 2026
STV-
ERMSE
0.5
0.7
0--
R Screz
0.-
02-
0.-
00
Budde Shangtuu
Sanf Popplng
Minebal
Inkso
Londin
Pari
Mescome
StuPado
Nasudii
Cape Torr
Sydaey
0.3
0.0
House
Image Number
(a)
(b)
Figure 6: (a) shows the results of the GDP prediction task across 13 different cities. (b) presents the results showing that satellite imagery has limited impact prediction. (c) demonstrates that increas- ing the number of street view images leads to progressive improvement in predictive performance.
We test House price, Public transport, and Drive ratio using Gemini-22.0-Flase under the Direct Metric Prediction setting, Contrary to prior findings that satellite imagery is often more discriminative than street view imagery for urban representation (Sun et al., 2025 Hao al. 2025). our results in Figure show that using street view images alone achieves performance comparable to using both street view and satellite imagery, and significantly outperforms using satellite imagery alone. This suggests that street view images provide more semantically rich and fine-grained visual cues, such as building façades, commercial signage, and infrastructure quality. These ground-level features are likely more tightly coupled with sociocconomic indicators and more readily interpreted by current LVLMs, which have been pretrained extensively on langeea pairs featuring such localized, human-centric content. While satellite imagery exhibits weaker predictive performance, it still contributes independent spatial context, such as urban morphology and building layout. However, may not offer the same level of semantic density at the resolution used in CityLens.
Effect of Street View Image Quantity To evaluate the impact of street view image quantity on prediction performance, we conduct experiments using Lama4-Maverice on the House Price task under the Direct Metric Prediction setting. Each region includes one satellite image and a varying number of street view images: 1, 5, 10, 15, or 20. We also test a no-image baseline following the design in Manvi et al. (2024b), where only the geographic coordinates and address are provided. In this setting, the model often refuses to respond, occasionally suggesting external resources like local housing websites. demonstrating both its conversational safety and limitation in open- world knowledge retrieval, From Figure bc| we observe a clear trend: increasing the number of street view images consistently improves model performance. This suggests that a richer visual context helps the model form a more accurate understanding of the region's ' socioeconomic condition.
3.4 REASONING CAPABILITIES AND MODEL ARCHITECTURE COMPARISON
Tasks
HP
PT
DR
BR
0.4
0.3
SiLLI-1ID
Gemmu3-12B
0.145
0.236
0.120
00.09
INIX 77
Gemma3-128-CoT
0.121
0.156
0.076
-0.049
0.2
Llamat-Maverick
0.459
Llamaa-MavvrrikkCoo
0,167
0.0
0.1
Gemini-200-laas
0.373
0.395
0.561
0.177
Gemtt--2.0-Flash-Coof
0.602 0.436 0.508 0.222
0.0
BR
MH
VC
PT
MH
LB
(a)
(b)
(c)
Figure 7: (a) Model performance with and without CoT prompting. (b) Performance of reasoning
models. (c) Comparison of different vision encoders used within the evaluated models.
Comparison of Shhain-of-Thougee Prompting vs. Standard Prompting Following the designs of Zhang et al. (2025b) and Xu et al. (2024) we implement a Chain-of-Thoupeee (CoT) prompting strategy tailored to the urban socioeconomic sensing context. The example CoT prompt is shown in A.7.4. We conduct an evaluation of CoT prompting on four representative tasks using three different models under the Direct Metric Prediction setting. As the Figure 7a below shows, we observe that the effect of CoT prompting varies by task. For the House Price task, CoT almost consistently improves performance across all models, suggesting it helps with the complex reasoning involved in interpreting housing couuing-relattd visual and semantic cues. In contrast, for the Drive Ratio task, CoT often reduces performance, possibly because this task relies more on direct visual features rather than step-by-step reasoning. From model perspective, Gemini-2.00 2.aaahh benefits most consistently
8

---

Published as a conference paper at ICLR 2026
Feature Extraction
Chain-of of Thought
Gemma3-48 4B
Gemman 128
Person: 3.0, Bikee 3.0) Heary Volice: 0.6 Light
Mental Health Ratio: 15%
1.0. 7.0. Wincow Oponing ig 0.0
Road: Sitewalk:
A: Btoo2 -Caplons........... ... -vecoal Baciozoonomic Profic:: Bascd on the images, the arca appcars be nixd incomo communiyy potendally virose ropiootioo The homes aparmment nginn. sugpesting g range beachfrott sportments or seatonal recktont component The commencill areas indiate some level of eoo coonomic activity but the the repair shop sugpests working clase presonce. The pr presonce of some disrapair and liter cotain aross su og potent eoonomic cna le-ges a les nosthetie benefis, but line and older infastructure are dharadterstic of estab ished urbon aress.
Greenery Treer S.O. O Greenery Grass & Strubs Sky: 7.0. Nature: 8.0
dlder,
tourtst
resources upkenp The peodriity the water a potential soset, attering
onal
<Calou belonoo Prowetta Strooooee: The csnatnuzioon saliityy powettl for sexional popu iilon fxctustion, and the presance cammerciel anans with potentio treli and nakee pollution coule increake strees wonks. Potent Pronection le Fachore The proaiiity the soner and ponsectin some green spaces could other opprrtuniliss mized-ncome neture smonbblly malgeting
negatioe
A: Person: 6.6, Bike: O.O. Honvy VoNicle 6.0, Light Window Road: Sidowalk: 8.5. Fummitur:2 2.G.
suggeets that meiients face economic challanges, which known in be associated with poorer mental health. The presence of cider housing stock and some signa of disnepair could indicate lower-nncom househalds. Casetal Nutrenbiity: The cometal location intmotuees patent charge,
Geesnery. Tree 3.0, Greenery Grass & Shrulss: 14. Sky 8.0. Nature 7,0
Caraadering fecters, beekwo provelence liedy sligttly higher than the aeenage, placing 18-25%
Figure 8: Representative error cases from Feature-Based Regression and CoT Prompting.
from CoT, with improvements across nearly all tasks. However, Llama4-Maverice shows performance drops with COT. One possible explanation is that Llama4-Maverias already possesses strong internal reasoning abilities, and the externally imposed CoT structure may not align with its learned inference patterns, leading to performance degradation.
Reasoning Model Performance with Standard Prompt We test three advanced reasoning models on three tasks that are inherently difficult predict directly from images under the Normalized setting. We note that CoT prompting and reasoning models represent two distinct approaches: CoT focuses on prompt design, injecting explicit reasoning steps into the input to guide model thinking, while reasoning models are evaluated using standard prompts to assess their intrinsic reasoning capabilities. As shown in Figure 7b) none of the powerful reasoning models achieve strong performance across all tasks, highlighting the challenge of predicting abstract social indicators from visual data alone. For instance, Iy-iiniin performs best on Violent Crime (At2=01455) but poorly on Bachelor Ratio, while 04-mini achieves the highest result on Bachelor Ratio (IR==0.465) but returns negative R2 on Violent Crime. Interestingly, we observe that Gemini-2.5-Flash performs better in its non- thinking version compared to its thinking version. This may be because urban socioeconomic sensing is not a purely logical reasoning task (as in math or code), but rather requires a nuanced combination of visual understanding and contextual inference. In such settings, reasoning-speciff adaptations may not always align well with the nature of the task.
The Impact of Different Vision Encoders We also follow the setup in Karamcheti et al. (0024) and conduct experiments using four models, which differ only in their vision encoders, under the Feature- Based Regression setting to investigate the role of visual backbones. This result in Figure suggests that LVLMs initialized with CLIP as the vision encoder produce most informative and semantically aligned outputs for urban socioeconomic sensing. The improved downstream performance may stem from CLIP's ability to extract visual cues that the language model can more effectively reason over. DINOv2 performs moderately but inconsistently, struggling on tasks like Life Expectancy, suggesting its self-supervised features may lack the semantic depth needed for abstract urban indicators. In contrast, SigLIP and INIK perform consistently poorly, indicating that ceneral-purposs contrastive or classification-brnt based encoders are less effective at capturing relevant visual cues.
3.5
ERROR ANALYSIS AND UPPER BOUNDS OF LVLMS
Error Cases on Challenging Tasks To better understand why LVLMs underperform on challenging tasks, we analyze representative errors observed under both the Feature- Based Regression and CoT Prompting setups. As illustrated in Figure S errors can arise from both visual perception and linguistic reasoning. For example, during feature extraction, Gemma3-4B fails to detect small but meaningful elements such as street signs, hallucinates non-existent persons, and underestimates visible greenery, assigning low score to "Grass & Shrubs". These errors reveal lack of fine tne-grained visual grounding and semantic alignment, which can propagate into downstream reasoning and prediction. We also observe reasoning errors in the CoT setting. For instance, in a Mental Health
9

---

Published as a conference paper at ICLR 2026
Table 2: Results of Fine-Tuned LVLMs on the CityLens Benchmark.
Tasks
GDP
Pop.
PT
DR
BH
MH
AH
BR
Fine-tuned Qwen2,5-VL.77B
0.628
0.231
0.502
0.628
0.872
0.418
0,364
0.442
Fine-tuned Qwen3-VL--8
0.626
0.107
0.545
0.638
0.869
0.397
0,304
0.536
Fine-tuned Llama3.2-VL-111B
0.562
0.287
0.348
0.256
0.829
0.248
0.256
0.157
prediction case, Gemma3 12B overly focuses on a few old and modest houses while overlooking numerous well-maintained, even upscale beachfront apartments visible in the street view images. Moreover, it fails to leverage the region's proximity to water, which is a known factor with strong aesthetic and calming effects that are directly linked to mental well-being. This suggests that current LVLMs may struggle to appropriately weigh holistic environmental cues during reasoning.
The Potential of Fine-Tunee LVLMs We conduct preliminary supervised fine-tuning on Qwen2.5- VL-7B, Qwen3-VL-8B, and Llama3.2 1.....2VV.. 2-VL-11B, using additional CityLens data not included in the benchmark under the Direct Metrie Prediction setup. Since the training data lacks samples for House Price, Violent Crime, and Life Expectancy, these tasks are excluded from evaluation. While general state-of-the-art LVLMs often perform poorly on CityLeen benchmark, frequently yielding near-zero even negative R2 scores, our results in Table show that fine-tuned LVLMs, regardless of base model or parameter size, achieve consistently strong performance across nearly all tasks. These findings highlight the promising potential of LVLMs for urban socioeconomic sensing, and further provide a preliminary estimate of the upper bound that such models can achieve when properly adapted for this domain. This reinforces the central motivation behind CityLens and underscores the value of developing Nomain-specinf LVLMs for addressing this socially important challenge.
4 RELATED WORK
Urban Socioeconomic Sensing A growing number of studies have attempted to predict socioeco- nomic indicators urban environments, Zhou et al. (2023) and Liu et al. (2023b) employ knowledge graph based approaches to support socioeconomic inference. Li et al. (2022) propose a contrastive learning framework based on structural urban imagery to support socioeconomic prediction. Fan cet al. (2023) extract features from street view imagery via a computer vision model to predict urban indicators. More recent studies have begun incorporating LLMs into this domain (Liu et al.) 2025
Ouyang al. 20244) Yan al. (2024) and HHo al. (2025) combine contrastive learning urban images with LLM-generated textual prompts. Manvi et al. (2024b) extracts geospatial knowledge from LLMs through fine-tuning and prompt design, while Li al. (2024b) evaluates LLMs socioe- conomic tasks across region-level and city-level granularity. Different from these works, CityLens the first benchmark to systematically evaluate the ability of LVLMs to predict socioeconomic indicators using both street view and satellite imagery.
recent years, LLMs have rapidly advanced in commonsense and reasoning. leading to the creation of diverse benchmarks across domains. These include bench-
Benchmarking LLM and LVLM
marks for code (Jimenez et al.) 2023 Jain et al. 2024). mathematics (Zhong et al. 2023; Wang
et al. 2023), city (Feng et al.) 2025a). well agent-based based tasks (Liu et al. 2023a] Qin al. 2023). b. Furthermore, numerous multimodal benchmarks have also emerged for LVLMs, including comprehensive benchmarks like Li et al. (003)) and Yue et al. (2024), and domain-specific ones such as Xia et al. (2024), Zhou et al. (2025) ) and Hu et al. (2024). While there are some benchmarks include urban imagery (c.g.. Feng al. (202cc by; Zhang et al. (2025a); Zhou al. (2025)). they are not specifically designed for urban socioeconomic sensing and address only a very limited subset of such tasks. Therefore, after comprehensive review of previous works, we propose a new benchmark to fill this gap and provide opportunities to bridge LVLMs with urban sensing applications.
5 CONCLUSION
In this paper, we introduce CityLens, a benchmark for evaluating the ability of large vision-language models to predict socioeconomic indicators from satellite and street view imagery. Through extensive experiments across 3 evaluation paradigms and 17 state-of-the-art models, we find that while current models exhibit promising perceptual abilities on certain visually grounded tasks, they still face major challenges in making accurate and generalizable predictions across domains and regions. CityLens provides a foundation for analyzing these limitations and motivates further research into enhancing the capabilities of large vision-language models in urban socioeconomic sensing.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work was supported in part by the National Key Research and Development Program China under Grant 2024YFC330760,, in part by the Guangdong Provincial Talent Program under Grant No.2023JC10X009 in part by the National Key Research and Development Program of China under grant 2024YFC3307603. This work also sponsored by Tsinghua- Toyota Joint Research Institute Inter-disciplinary Program.
ETHICS STATEMENT
Privacy All street view images CityLens are sourced from platforms such as Google Street View, Baidu Maps, and Mapillary, which enforce automatic blurring of sensitive visual information, including faces and license plates. No personal or identifiable data is collected, stored, or annotated by the authors. All images are used exclusively for academic research purposes. Furthermore, the image resolution is coarse-grained, and all imagery is de-identified, ensuring that no individual-lever visual content is exposed. The socioeconomic indicators used the benchmark are aggregated the regional level (e.g.. Census Tract) rather than the individual level, further mitigating privacy risks.
Geographic Bias and Fairness CityLens covers 17 cities across all six continents, ensuring a high level of geographic diversity. However, for certain indicators, particularly those culturally sensitive data, ground-truth labels are unavailable in some cities, most notably in regions within the Global South. As result, the distribution of prediction tasks is uneven across regions, which may raise concerns of underrepresentation of Global South cities. Beyond task availability, such geographic imbalances data coverage may also contribute to potential geographic biases in how current LVLMs generalize across diverse socioeconomic and cultural contexts. We include a preliminary bias audit in Appendix A.9 where we observe noticeable differences in performance across cities. We highlight this both diagnostic insight and an opportunity for future research, particularly in addressing fairness and robustness in cross-regional prediction scenarios.
Misuse Declaimer CityLens is designed solely for research and evaluation purposes. It should not be used to inform real-world decisions in areas such as policing, health, or public resource allocation. The benchmark includes sensitive sociocconomic and crime-related indicators that highly context-dependent. Any use of model outputs evaluated on CityLens for operational or policy purposes must be preceded by ethical review, fairness assessment, and domain-speciff validation. We strongly discourage the use of this benchmark for surveillance or enforcement without appropriate safeguards.
REPRODUCIBILITY STATEMENT
Our work aims to ensure transparency and reproducibilityy through full release of both data and code. All resources made publicly available via hilh:tss://gtthtmi com www.w//ontttnaaaeeeeeeeeeeeeeaat /CityLens
fib- lab
ata
Dataset
The CityLen Benchmark includes two versions of the dataset:
CityLens Gioogle/Baidubaasedee This version uses street view images retrieved via Google and Baidu APIs. Due to licensing restrictions, we cannot directly distribute the images. Instead, we follow the practice T'n et al. (2023); Huang et al. (024)); Wang et al. (2025), and provide complete list pano ids along with download scripts for automated image retrieval. Additionally, we release the task data covering 11 sociocconomic indicators in the repository.
SttyLennsMapillar (Open-source): This version leverages street view images from Mapillary, an open-source street-level imagery platform. All images in this version are fully accessible.
Both versions of the dataset share the same set of satellite images.
Code We release all code required for data processing and evaluation. Detailed instructions and usage examples provided in the README.nd file within the repository.
11

---

Published as a conference paper at ICLR 2026
REFERENCES
AbdeIrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras. arXiv preprint arXiv:25033.1743, 2025.
Josh Achiam, Steven Adler, Sandhini Agarwal, and et al. Gpt-4 technical report. arXiv preprini arXiv:2303.08774, 2023.
Meta AI. Ilama4. http:://www...aaa...coo///o///// 2025a.
Mistral AI. Mistral httppp/s//matttaaaaaaaa/////sasa
2025b.
Zhiqiang Shen Aidar Myrzakhan, Sondos Mahmoud Bsharat. Open- From multi- choice to open-style questions for Ilms evaluation, benchmark, and arena. arXiv preprint AXiv:22006005545 2024.
Amazon. Amazon nova. http://////...aaaaa...o///////a. 2025.
Shuai Bai, Keqin Chen, and Xuejing et al Liu. Qwen2.5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
Chicago. Crimes 2001 present.
ety/Crimes-20111 99//////.. 8m zvie 2019. Accessed: 2021.
Google DeepMind. Gemini2.0,
/ v2 2025.
Zhuangyuan Fan, Fan Zhang. Becky P. Y. Loo, and Carlo Ratti. Urban visual intelligence: Uncovering hidden city profiles with street view images. Proceedings of the National Academy of Sciences, 120(27)xe2204111122 2023. doi: 10. 2223011
Jie Feng, Tianhui Liu, Yuwei Du, Siqi Guo, Yuming Lin, and Yong Li. Citygpt: Empowering urban spatial cognition of large language models. In Proceedings of the s1 ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, KDD 25, pp. 591-602, New York, NY, USA, 2025a Association for Computing Machinery. ISBN 979840071 11477444442 4542. doi: 10.1 10.1145/371 18963371189633368777
URL httts::t//ddte /tt.////oooooooooo//oo/oo/////// wrg/10.....o.oooooooooo..o...1.. 1145/1711111
37 3736878
Jie Feng, Shengyuan Wang, Tianhui Liu, Yanxin Xi, and Yong Li. Urbanllava: A multi-modal large language model for urban intelligence. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6209-6219, 2025b.
Jie Feng, Jun Zhang, Tianhui Liu, Xin Zhang, Tianjian Ouyang, Junbo Yan, Yuwei Du, Siqi Guo, and Yong Li. Citybench: Evaluating the capabilities of large language models for urban tasks. In Proceedings of the s1 ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pp. 5413-5424, 2025c.
Zhenyu Han, Tong Xia, Yanxin Xi, and Yong Li. Healthy cities: A comprehensive dataset for environmental determinants of health in england cities. Scientific Data, 10(1):165, 2023, doi: 1011036441597-003-0006663
Xixuan Hao, Wei Chen, Yibo Yan, Siru Zhong, Kun Wang. Qingsong Wen, and Yuxuan Liang. Urban- vlp: Muati-granyyarrty vasion-language pretraining for urban socioeconomic indicator prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 28061-28069, 2025.
Ce Hou, Fan Zhang, Yong Li, Haifeng Li. Gengehen Mai, Yuhao Kang, Ling Yao, Wenhao Yu, Yao Yao, Song Gao, et al. Urban sensing in the era of large language models. The Innovation, 6(1), 2025.
Xueyu Hu, Ziyu Zhao, Shuang Wei, Ziwei Chai, Qianli Ma, Guoyin Wang, Xuwu Wang, Jing Su, Jingjing Xu, Ming Zhu, et al. Infiagent-dabentn Evaluating agents on data analysis tasks. arxiv preprint arXiv:2401.0554 2024.
12

---

Published as a conference paper at ICLR 2026
Tianyuan Huang, Zejia Wu, Jiajun Wu, Jackelyn Hwang, and Ram Rajagopal. Citypulse: fine- grained assessment of urban change with street view time series. AAAI Press, 2024. ISBN 978-1-57735-887-99 URL https://ati..oooooo orw.:o..ggo..oo 1609/ /wt///0//.////////............a. v38120. 30216
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Panjia Yan, Tianjun Zhang, Sida Wang. Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.0793 2400444747774 2024.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofrr Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint
arXiv:2310.06770, 2023.
Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vlms: investigating the design space of visually-conditioned language models. In Proceedings of the s1 International Conference on Machine Learning, ICML'24. JMLR.org. 2024.
Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, al. Minimax-01: Scaling foundation models with lightning attention. arXiv preprint 250:::2501008111.. 2025.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench, Bench- marking multimodal Ilms with generative comprehension. arXiv preprint arXiv:2307.16123 2023.
Tong Li, Shiduo Xin, Yanxin Xi, Sasu Tarkoma, Pan Hui, and Yong Li. Predicting multi-level socioe- conomic indicators from structural urban imagery. In Proceedings of the 31 ACM international conference on information & knowledge management, pp. 3282-3291, 2022.
Wangyue Li, Liangzii Li, Tong Xiang, Xiao Liu, Wei Deng, and Noa Garcia. Can multiple-choice questions really be useful in detecting the abilities of Ilms? In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation 2024a.
(LREC-COLING 2024), pp.
Zhuoheng Li, Yaochen Wang, Zhixue Song, Yuqi Huang, Rui Bao, Guanjie Zheng, and Zhenhui Jessie Li. What can Ilm tell us about cities? arXiv preprint (:Xiv:24411.179)) 2024b.
Yuming Lin, Xin Zhang, Yu Liu, Zhenyu Han, Qingmin Liao, and Yong Li. Long-term detection and monitory of chinese urban village using satellite imagery. Proceedings of the Thirty-Third International Joint Conference Artificial Intelligence, 2024. ISBN 978-1-956792.0041. URL
http:::/ddo...ggg//// 2ww.11....43... hwwp:446633//a.2222222.. 2024/813
Tianhui Liu, Hetian Pang, Xin Zhang, Jie Feng, Yong Li, and Pan Hui. Cityrise: Reasoning urban socio-economic status lainuala models via reinforcement learning. arXiv preprint arXiv:2510.22282, 2025.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308. 0338883300.06688. 2023a.
Yu Liu, Xin Zhang, Jingtao Ding, Yanxin Xi, and Yong Li. Knowledge-infusee infused contrastive learning for urban imagery-based socioeconomic prediction, 2023b.
Rohin Manvi, Samar Khanna, Marshall Burke, David Lobell, and Stefano Ermon. Large language models are geographically biased, 2024a.
Rohin Manvi, Samar Khanna, Gengehen Mai, Marshall Burke, David B. Lobell, and Stefano Ermon. Geollm: Extracting geospatial knowledge from large language models. The Twelfth International Conference Learning Representations, 2024b. URL https: http://ppeeeevieeee net. foru
Pid=TqL2XBWXPP3
United Nations. Transforming our world: the 2030 agenda for sustainable development. United Nations, 2015. URL (tllhtttss:... /tsttts://sds....
Accessed: 2024-12-03.
13

---

Published as a conference paper at ICLR 2026
New York. Citywide crime statistics incident level data. https: wwtp://wwwww.m http:://wwww.cc.o///////0 Accessed: 2021.
2019.
Tomohiro Oda and Shamil Maksyutov. ODIAC fossil fuel CO2 emissions dataset (ODIAC2022), 2015. ott.o//0..aaaaaaaaaa.aa.aaaeaaar
OpenAl. gpt-4.1. httpss//ggena...cmm/nee/e///s
2025.
Tianjian Ouyang, Xin Zhang, Zhenyu Han, Yu Shang. and Yong Li. Health clip: Depression rate prediction using health related features in satellite and street view images. In Companion Proceedings of the ACM Web Conference 2024, pp. 1142-1145, 2024.
Martino Pesaresi and Panagiotis Politis. GHS-BUILT-H R2022A- - GHS building height, derived from AW3D30, SRTM30, and Sentinel2 composite (2018) OBSOLETE RELEASE, 2022. URL
Dataset.
PLACES. Local data for better health. http:://ww..ccc.o///bla. 2025-05-11.
Accessed:
Lianjia Real Estate Platform. Lianjia housing price data. htttps://iaania..
2020.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, al. Toolllm: Facilitating large language models master 16000+ real-word apis. arXiv preprint arxiw:2307.17789 2023.
SafeGraph. http://http::://w...eeee
com/
San Francisco. Police departmment i reports historical 2003 to present. https://data.. Egov (rcor.oogg/uullcc--fffet////.-- Deparrttee-ttttttttteteeeee-e-e. Incident -eporrtsttep-tttt- yvryj 2019. Accessed: 2021.
istorical-2003/mme
Fengze Sun, Yanchuan Chang, Egemen Tanin, Shanika Karunasekera, and Jianzhong Qi. Flexireg: Flexible urban region representation learning. In Proceedings the 31 ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, pp. 2702-2713, Association for Computing Machinery, 2025. ISBN 9798400714542. URL http::ttt///ttm.. htttts:///oi..ooo http://adl.rrr1111111//////31131 3736965
National Household Travel Survey. httts::/www.bbs
2017.
Andrew J Tatem. Worldpop, open data for spatial demography, Scienti Data, 40):1-4,2017.
Gemma Team. Gemma 2025. htttpe///ariixx.orr/////22333
Lei Wang, Martin Kada, Tianlin Zhang, and Jie He. Cross-platform complementarity: Assessing the data quality and availability of google street view and baidu street view. Transactions in Urban Data, Science, and Technology, 4(1):22-47, 2025, doi: 10. 10.1177/7541231121111
Tingting Wang and Fubao Sun. Global gridded gdp data set consistent with the shared socioeconomic pathways. Scientific Data, 9:221, 2022. doi: 10. 10.10338s41157-02--01133
Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang. Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level scientific problem-solving abilities of large language models. arXiv preprint arXiv:2307.10633 2023.
Daniel J. Weiss, Andrew Nelson, Carlos A. Vargas-Ruiz, et al. Global maps of travel time to healthcare facilities. Nature Medicine, 183118888888 2020. doi: 10. 10.1038/441591-020 1059-1.
URL UBL.
Peng Xia, Ze Chen, Juanxi Tian, Yangrui Gong. Ruibo Hou, Yue Xu, Zhenbang Wu, Zhiyuan Fan,
Yiyang Zhou, Kangyu Zhu, et. al. Cares: A comprehensive benchmark of trustworthiness in medical vision language models. Advances Neural Information Processing Systems, 37:140334-14000 2024.
14

---

Published as a conference paper at ICLR 2026
Guowei Xu, Peng Jin, Hao Li, Yibing Song, Lichao Sun, and Li Yuan. Llava-cot: Let vision language models reason step-by-step. 2024. URL Btllhttt///::iiv/ org/ abs/ 10440
Yibo Yan, Haomin Wen, Siru Zhong, Wei Chen, Haodong Chen, Qingsong Wen, Roger Zimmermann, and Yuxuan Liang. Urbanclip: Learning text-enhanced urban region profiling with contrastive language-image pretraining from the web. In Proceedings of the ACM Web Conference 2024, pp. 4006-4017 2024.
Xixian Yong and Xiao Zhou, Museel: Predicting urban socioeconomic indicators via multi-semantic contrastive learning. arXiv preprint arXiv:2407.09553, 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal under- standing and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9556-9567, 2024.
Xin Zhang, Yu Liu, Yuming Lin, Qingmin Liao, and Yong Li. Uv-sam: Adapting segment anything model for urban village identification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 22520-22528, 2024.
Xin Zhang. Tianjian Ouyang, Yu Shang, Qingmin Liao, and Yong Li. UrbanMLLM: Joint learning of cross-view imagery for urban understanding. 2025a. URL bxtpps://penneeievw.eeee
orum?id=YBhi 9Vp
Yunke Zhang, Ruolong Ma, Xin Zhang, and Yong Li. Perceiving urban inequality from imagery using visual language models with chain-of-thought reasoning. In Proceedings of the ACM on Web Conference 2025, pp. 5342-5351. Association for Computing Machinery, 2025b. ISBN doi.
9798400712746. URL /o/g////////....////11.. 11w.11..111133341.... 3714536,
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang. Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centra benchmark for evaluating foundation models. arXiv preprint (xXXx:2334.063644 2023.
Baichuan Zhou, Haote Yang, Dairong Chen, Junyan Ye, Tianyi Bai, Jinhua Yu, Songyang Zhang, Dahua Lin, Conghui He, and Weijia Li. Urbench: A comprehensive benchmark for evaluating large multimodal models in multi-view urban scenarios. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 10707-10715, 2025.
Zhilun Zhou, Yu Liu, Jingtao Ding, Depeng Jin, and Yong Li. Hierarchical knowledge graph learning enabled socioeconomic indicator prediction location-based social network. In Proceedings of the ACM Web Conference 2023. Association for Computing Machinery, 2023.
Zillow. Housing data. ht www www.ziiliiioo.. how.oo/.ccommreeccccccaaaaaalll 2020.
15

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 THE USE OF LARGE LANGUAGE. MODELS
This paper made limited use of large language models to assist with improving the clarity and stylistic quality of writing. The use of L.MS was restricted to language editing and polishing of author-written content. No text was generated solely by the model without human verification, and all technical claims, experimental results, and interpretations were entirely conceived, written, and validated by the authors. The authors retained full responsibility for the content of the paper and ensured that the use of LLMs complied with relevant ethical standards and publication guidelines.
A.2 DISCUSSION
Our benchmark results highlight both the potential and limitations of using LVLMs to predict region- level socioeconomic indicators. While some tasks, particularly those with visually salient correlates such as building height achieve moderate performance, most indicators remain challenging to estimate accurately. Their outputs often converge around city-level averages, suggesting that the model lacks sensitivity to intra-city variation, and consequently exhibits limited geospatial grounding. In the two observe that different tasks tend to favor different strategies: some
LVLM as predictor paradigms, perform better under direct value prediction, while others benefit from normalized estimation. Overall, our results indicate that the Feature-Based Regression paradigm, where the LLM functions feature enhancer, significantly outperforms the two predictor-based methods. These findings suggest several promising directions for future research. First, while the reature-based method relies on trained LASSO regressor, the predictor-based methods are initially evaluated in a zero-shot setting. Through fine-tuning LVLMs, we have demonstrated the significant potential of training domain-specific LVLMs for urban socioeconomic sensing. Second, future improvements may come from designing prompts that more closely reflect human reasoning patterns, beyond standard CoT prompting. While our current experiments already incorporate CoT prompts, we believe that further performance gains may be achieved through cognitively informed prompt design. To support this direction, also outline hypothesis in Appendix AA.10jnn how LVLMs accomplish urban socioeconomic sensing. Finally, we envision the development of a domain -spaiin- agent framework tailored to urban socioeconomic sensing, which could combine visual perception, geospatial knowledge, and reasoning modules to make robust and context-aware predictions in real-world scenarios.
A.3 BENCHMARK RESULTS WITH MAPILLARY STREET VIEW IMAGES
To promote transparency, completeness, and reproducibility, we construct an alternative version of CityLLnns using publicly available street view images from the open-source Mapillary platform, referred to as CityLens-Mappan benchmark and is intended for use in scenarios where proprietary street view APIs (Google and Baidu) are inaccessible, representing worst-case scenario due to access restrictions or licensing limitations.
This version serves as open-source complement to the main
Due to the relative sparsity and inconsistency of open-source street view coverage, the number of supported tasks in CityLens-Mapilla is reduced. Table B summarizes the number of available prediction tasks for each indicator, comparing the original CityLens with the Mapillary-based version. Table presents the results on CityLens-Maiillar. We observe modest drop performance compared to the original CityLens, which we attribute to the generally lower image quality and coverage of the Mapillary platform. For instance, in the UK region, predictions for Life Expectancy show several negative R² values, likely resulting from degraded visual input. While we applied light manual filtering to remove extremely poor-quality images, some noise remains due to the inherent limitations of open-source data. Nevertheless, performance trends on Cityl Mityllea--aapllary closely mirror those on the original dataset, demonstrating that open-source platforms can still provide strong utility for socioeconomic prediction. These findings highlight the viability and promise of open-source alternatives like, especially for future research in urban socioeconomic sensing.
A.4 NONLINEAR REGRESSOR RESULTS UNDER FEATURE- BASED REGRESSION SETTING
We further conduct additional experiments using nonlinear regressors, including Random Forest, XGBoost, and ML.,, regress on the 13-attribute vectors extracted from Gemma3-27B and Llama4-
16

---

Published as a conference paper at ICLR 2026
Table 3: Number of valid prediction tasks per indicator across regions in CityLens vs. CityLens- Mapillary.
Task
Google/Baidu
Mapillary
GDP
1000
805
Population
1000
824
House Price
777
527
Public Transport
500
409
Drive Ratio
500
Mental Health
500
416
Accessibility to Health
1000
809
Life Expectancy
193
89
Building Height
1000
813
Vialent Crime
396
345
Bachelor Ratio
500
382
Table Main Results on Peature-Bassee Regression method from Mapillary data.The values in the table represent R² scores. "Mean" denotes the average performance across tasks, and "SD" refers to the standard deviation. each row, bold indicates the best result, and underline denotes the second-best
Domain Tasks
Econ.
Crime
Trans.
Env.
Health
Edu.
Overall
Pop.
HP
DR
Gemma3-4B
0.132
0.100
0.013
0.274
0.170
0.092
0.201
0.153
G2mma3-12B
0.453
0.148
0.146
0.020
0.348
0.242
0.573
0.039
0.342
0.174
0,061
0.231
0.171
Gemma3-277
0.471
0.188
0.165
0.016
0.299
0.274
0.583
0.074
0.353
0.205
0.088
0.247
0.165
Qwen2.5VL-3B
0.406
0.142
0.206
0.007
0.228
0.199
0,563
0.028
0.365
0.126
0.040
0,210
0,166
Qwen2.5VL--BB
0.401
0.161
0.150
0.056
0.376
0.205
0,554
00.189
0.357
0.043
0.160
0,207
0.196
Qwen2.5VL-32B
0,408
0.148
0.220
-0.018
0.329
0,220
0.563
-0.044
0.341
0.085
0.107
0.214
0.176
Llama4-Scout
0.401
0.161
0.053
-0.010
0.295
0,288
0,570
0.078
0.361
-0.0004
0.167
0,215
0.176
Llama4-Maverick
0,470
0.171
0.157
-0.007
0.287
0.362
0.594
0.082
0.357
-0.0004
0.088
0.233
0.188
Mistral-small-3.1222
0.462
0.175
0.109
0.021
0.376
0.281
0.560
0.112
0.366
00,0004
0.099
0.233
0.179
Phi-44multimodal
0.017
0.308
Nova-lite-vl
0.424
0.131
0.163
0.002
0.273
0.229
0.533
0.046
0.292
-0,0004
0.116
0.201
0,163
Minimax-01
0.433
0.139
0.206
-0.016
0,362
0.250
0.587
0.030
0.374
-0.0004
0.128
0.227
0.186
Gemini-220-FHi
0.443
0.157
0.130
-0.002
0.444
0.326
0.607
0.093
0.357
0.059
0.111
0.248
0.187
Gemini-2.5-Flush
0.452
0.210
0,198
0.040
0.362
0,246
0.598
0.004
0.297
-0.0004
0.085
0,227
0.183
GP--4o-mini
0.394
0.135
0.151
0.007
0.273
0.371
0.533
0.071
0.384
0.0004
0.071
0.217
0.173
GPT--.I-mini
0.185
0.092
0.019
0.350
0.159
0.304
0.108
GPT--41-namo
0.429
0.179
0.037
0.005
0.347
0.227
0.520
0.112
0.332
0.025
0.113
0.211
0.166
Maverick. The results are summarized in the Table 5 While nonlinear regressors offer modest improvements over linear models in certain tasks such as GDP prediction, they do not consistently outperform across tasks. In particularly challenging indicators like Violent Crime and Mental Health, R scores remain low or even negative for all nonlinear regressors, suggesting that the primary bottleneck is not the regression capacity, but rather limitation in the expressiveness of the 13-attribute vectors extracted by current LVLMs.
Table 5; Results of nonlinear regressors, including Random Forest, XGBoost, and MLP.
Method
Model
GDP
Pop.
HPP
VC
PT
DR
BH1
MH
AH1
LE
BR
RF
Gemma3-27B
0.563
0.234
0.106
-0.335
0.477
0.587
0.618
0.181
0.302
0.240
0.256
RF
Llama4-Scout
0.516
0.239
0.114
-0.063
0.443
0.448
0.575
0.207
0.304
0.176
0.266
XGBoost
Gemma3-278
0.507
0.263
0.130
-0.533
0.357
0.392
0.571
0.199
0.244
0.129
0.174
XGBoost
Llama4-Scout
0.549
0.153
0.210
-0.072
0.411
0.370
0.556
0.180
0.294
0.112
0.161
MLP
Gemma3-27B
0.467
0.332
0.116
0.085
0.507
0.486
0.599
-0.053
0.296
-10.632
0.218
MLP
Llama4-Scout
0.413
0.247
0.192
0.005
0.317
0.402
0.610
-0.155
0.309
-1.042
0.102
17

---

Published as a conference paper at ICLR 2026
A.5 DETAILS ABOUT CITYLENS DATASET
A.5.1 SUMMARY OF INDICATORS IN CITYLENS
We provide an overview of all indicators considered in the construction of the CityLens benchmark. Table B lists all 28 collected indicators, along with their data sources and whether they are ultimately selected as prediction tasks.
Economy Under the economy domain, we cover 7 critical indicators: GDP, house price, population, median household income, poverty 100%. poverty 200% and income Gini coefficient. For GDP, we utilize a global dataset that provides GDP estimates with a spatial resolution of 1 km Wang & Sun (2022). For population, we adopt estimates from WorldPop [Taemm] (0117), global demographic dataset with km spatial resolution that provides consistent population counts actoss countries. For house price, we collect data from multiple sources tailored to each country's s context: (1) For US cities, the Zillow Home Value Index (ZHVI) (Zillow 2020), available the 7IP code level. We map these values to census tract boundaries using spatial overlays, enabling fine-grained local prediction. (2) For UK cities, we target the Middle Layer Super Output Area (MSOA) level, obtaining house price data from Han et al. )2023). (3) For Chinese cities, we collect house price data from LianJia (Platform 2020), one of China's s largest online real estate platforms. For median household income, poverty 100%, and poverty 200%, we obtain the raw data from (SateGraph] We obtain the ground-truth values for the income Gini coefficient from Zhang et (2025b).
Transport In the transport domain, we include seven indicators: PMT, VMT, PTRP, VTRP, walk and bike ratio, public transport ratio, and drive ratio, following the design of Fan et al. (2023). The underlying data sourced from nom[Syyvey] (2017), which provides commuting behavior statistics the census tract level across the United States. Table 7 outlines the definitions of the seven transport related indicators.
Crime In the crime domain, we focus on two indicators: violent crime incidence and non-violent crime incidence, both defined the number of crime occurrences per census tract. The data is collected from the official websites of individual US cities (Chicago 2019 New York 2019 San Francisco 2019). which publish annual crime reports and geolocated incident-level data.
Health For the health domain, we include 9 kinds of indicators capture different dimensions of urban health outcomes: obesity, diabetes, cancer, no leisre-time physical activity (LPA), mental health, physical health, depression rate, accessibility to healthcare, and life expectancy. The first tasks focus on the United States only, using data from "Local Data for Better Health" (PLACES). The Accessibility to Healthcare task is defined globally, using dataset that quantifies walking-onf, travel time to the nearest healthcare facility (Weiss et al., 2020). The Life Expectancy task targets the United Kingdom, where we use data from Han et al. (2023) to obtain male life expectancy estimates at the MSOA level.
Environment In the environment domain, consider two indicators: Carbon Emissions and Building Height. For carbon, we use global estimates from Oda & Maksyutov (2013). We use global building height data obtained from Pesaresi & Politis (022)) which provides global coverage at spatial resolution of 100 meters.
Education Following Liu et al. (023b), we use the Bachelor Ratio, defined as the proportion of residents holding bachelor's ss degree higher, as the target variable the education domain. The ground-trute data for this indicator is obtained from SafeGraph (SafeGraph), which provides fine-grained demographic datasets across the United States.
A.5.2 DATA MAPPING AND AGGREGATING
In CityLens, each region serves as a unit of prediction and is represented by 1 satellite image and 10 street view images. For each region, associate one scalar label for the target indicator by mapping and aggregating raw tabular data from heterogeneous sources. Regarding the label mapping and aggregation strategies:
18

---

Published as a conference paper at ICLR 2026
Table Summary of 28 indicators in CityLens.
Domain
Indicator
Source
Selected
GDP
Wang & Sun
2022
House Price
Zillow (2020
Han et al. 2023
Platform
Economy
Population
Tatem 2017)
Median Income
SaleGraph
Poverty 100%
SafeGraph
Poverty 200%
SaleGraph
Income Gini Coefficient
Zhang ct a.
(025b)
Education
Bachelor Ratio
SafeGraph
Chicago 2019
New York 2019
Violent
San Francisco
2019)
Crime
Chicago 2019
New York
2019
Non-Violent
San Francisco
2019
PMT
Survey
2017
VMT
Survey
2017
PTRP
Survey
Transport
VTRP
Survey
2017
Walk and Bike
Survey
2017
Drive Ratio
Survey
2017
Public Transport
Survey
2017
Obesity
PLACES
Diabetes
PLACES
LPA
PLACES
Cancer
PLACES
Health
Mental Health
PLACES
Physical Hcalth
PLACES
Depression
PLACES
Life Expectancy
Han el al. (2033
Accessibility to Healthcare
Weiss al. (020)
Carbon Emissions
Oda & Maksvutov (0015)
Environment
Building Height
Pesarest & Poliui 2022
Table 7: Definitions of the seven indicators in the Transport domain.
Topic
Indicator
Label
% (onollation/16)ccommmi by driving alone Estimated personal miles traveled working weekday
Drive Ratio
PMT
working weekday Estimated vehicle miles traveled on a working weekday Estimated vehicle trips traveled on a working weekday % (ropulation(~16ccomme by public transit % (PPoulation(~11)oommm) by walking and biking
Estimated personal trips traveled
PTRP
Transport
VMT
VTRP
Public Transit
Walk and Bike
For Public Transport Ratio, Drive Ratio, Mental Health, and Violent Crime, we follow the methodology in Fan et al. (2023). The data provided at the CT level, we CT as the key to directly map indicators to regions. For Bachelor Ratio, the original data also available at the CT level, so we apply the same CT-based mapping.
For House Price:
In the U.S., the original values are provided by Zillow at the ZIP code level. Using the official crosswalk between ZIP codes and CTs, we assign ZIP-level values CTs. Since some CTss and ZIP codes do not align perfectly, we use averaging in cases of overlap.
In China, the raw dataset consists of individual records with latitude and longitude. We aggregate values by averaging all records falling within each satellite image's coverage.
19

---

Published as a conference paper at ICLR 2026
In the U.K., the data is already available the MSOA level, perform direct MSOA-to- region mapping.
For Life Expectancy, we use the same AISOA-based strategy.
For globally available GDP, Population, Building Height, and Accessibility to Healthcare, the original data is provided in GeoTIFF format. For each satellite image, we extract the values covering the image's geographic extent and compute the average the region-level indicator.
A.6
ADDITIONAL. INFORMATION ABOUT THREE EVALUATION METHODOLOGIESS
A.6.1 PROMPT DESIGN AND CASE ANALYSIS
We provide additional insights into the design and behavior of LVLMs under the three evaluation methodologies. Table & summarizes the representative prompts used in the three paradigms, high- lighting their structural differences and role-setting strategies. Figure shows a case in the Direct Metric Prediction setting, where the model refuses to estimate GDP due insufficient information. Figure 10 shows an example under the Normalized Estimation setting, where the model is asked to predict the bachelor ratio based on regional images. Figure II presents an example from the Feature- Based Regression method, where the model scores a street view image along predefined urban visual attributes to support downstream prediction.
Table 8: Prompt comparison across the three evaluation methodologies in transport domain tasks.
Method
Prompt
Direct Metric Prediction
Suppose you professional transport data analyst in (city), [country] Based on the provided satellite imagery and several street view photos, please estimate 'the [indicator]' the census tract where these images are taken. Consider factors such as road infrastructure, visible traffic pat- terns, availability of public transport options, pedestrian walkways, and any other relevant details that might influence these transport behaviors in the area.
Please provide single specific number (not a range or approximate value) for "[ [iindicatrr''. No explanation is needed. needed.Example answer: (example num]
Normalized Metric Estimation
Suppose you are professional transport data analyst in (city), [country] ). Based on the provided satellite imagery and several street view photos, please estimate 'the (indicator) in the are taken. Consider factors such as road infrastructure, visible traffic pat- terns, availability of public transport options, pedestrian walkways, and any other relevant details that might influence these transport behaviors in the
tract where these images
Please provide single specific number for "(indicator'" (on scale from 0.0 to 9.9). No explanation is needed. Example answer: 8.8.
Feature- Based Regression
Analyze the provided street view image. For each of the following indicators, provide a score from 0.0 to 9.9 representing its presence or prominence in the image. The output should only be the indicator name followed by its score, one indicator per line. No need for explanations additional text.
Indicators: Person; Bike; Heavy Vehicle; Light Vehicle; Façade; Window
& Opening: Road; Sidewalk: Street Furniture; Greenery Tree; Greenery
Grass & Shrubs; Sky; Nature
Example: Person: 2.5; Bike: 0.0;
A.6.2 SUPPLEMENTARY COMPARISON OF THREE EVALUATION METHODOLOGGESS
We present two supplementary figures in Figure 12
comparing Feature-Based Regression with Direct Metric Prediction, and the other with Normalized Estimation. These visualizations enable side-by-side assessment of performance across all socioeconomic indicators.
20

---

Published as a conference paper at ICLR 2026
Suppose you are professional socioeconomic data analyst CapeTown. Based on the provided satellite image and several street view photos taken within the same area covered by the satellite image, please estimate 'the average (the unit PPP 2005 international dollars) for this building structures, road infrastructure, visible traffic patterns, use characteristics, greenery, and any other relevant features, <itaeeee:eeiaamaeeeeeiieeeee
Q:
Consider factors such
Please provide single specific number (not range approximate value) for 'average GDP in this area. No explanation needed. Example answer: 1914240768
Groundtruth: 40729580.0
A(GPT4.1- "Sorry, cannot provide an estimate the average GDP for the area based solely on these images."
Figure 9: An example of model refusal in the Direct Metric Prediction setting.
Satellite Image and Street View Images in SanFrancisco
Q: Suppose you demographic analyst specializing education statistics in SanFrancisco, United States. Based the provided satellite imagery and several street view photos, please estimate bachelor ratio the census tract where the images are taken. Consider factors such as location, visible property features, neighborhood condition, and any other relevant details. Please provide single specific bachelor ratio number (not range or approximate value),, expressed as a decimal between 0 and 1. No explanation is needed. Example answer: 0.46
Groundtruth: 0.62
A(Gemma3-128): 0.62
A(Gemma3-49) 3.2
A(Gemma3-278): 0.38
A(Amazon-nova-lle--rii 0.43
A(Gemini-2.0-ffen 0.32
A(Gemini-2.5-flash) 0.58
A(Llama4-Maverick)): 0.43
A(Llama4-Scout): 0.34
A(GPT4.1-mini): 0.27
A(GPT4.1-nano) 0.34
Figure 10: Case example for Normalized Metric Estimation.
From the figures, evident that the Feature-Basse approach, where the large vision-language model acts as feature enhancer, consistently outperforms the Direct and Normalized approaches, in which the model is expected to behave as a numerical predictor. This suggests that current LVLMs, while powerful in perceptual and language tasks, are still more effective when used to extract structured visual representations rather than to directly generate precise socioeconomic estimates. Although large vision vision-language models have made impressive strides, accurately predicting fine- grained, region-level socioeconomic indicators remains highly challenging, highlighting the need for further advancements.
21

---

Published as a conference paper at ICLR 2026
image. each following 13 indicators, provide score from 0.0 9.9
Analyze the provided
representing its presence or prominence in the image. The output should only be indicator name followed by score, one indicator
Indicators:
Person
Bike
Heavy Vehicle
Light Vehicle
Façade
Window & Opening
Road
Sidewalk
Street Furniture
Greenery - Tree
Streat View Image Chicago,US
Greenery Grass Shrubs
Sky
Nature
Example:
Person: 2.5
Bike: 0.0
A(Gemmaa-12B) Person: 0.0'n Bike: 0.0ln Heavy Vehicle: 3.5in Light Vehicle: 2.0In Façade: 6.5mn Window & Opening: 3.0in Road: 7.01m Sidewalk: 6.0in Street Furniture: 1.0hn Greenery Tree: 9.0In
Greenery Grass & Shrubs: 7.0ln
Sky: 8.0in
Nature: 9.0
Figure 11: Prompt template for guiding large vision-language models extract 13 visual features from a street view image.
Festure ite
Festure Rt
0.2
0.2
0.0
0.0
House Price
Menni Health Expestancy
-02
-02
Bacheke Raño
Acces to Healheae
Building Height
Drive Ramo
Direct R
Normalized R
(a)
(b)
Figure 12: (a) shows the performance comparison between Feature- Teature-Based Regression and Direct
Metric Prediction. (b) compares Feature-Baan -ased Regression with Normalized Estimation..
A.7 ADDITIONAL EXPERIMENTAL SETUP DETAILS
A.7.1 LVLMS
We consider a diverse set of LVLMs as baselines to benchmark our proposed methods. The selected models include both open-source and proprietary systems, covering range of model sizes and capa- bilities. We choose Gemma3 1ammma3--B/1i 1B/141//1221 12BB2BB [Team] 2025), Swwn2.5VLL3BB77BB
(Bai et al. 2025 (Al 2025b), Phi-4-multimoaal (Aboue
Haama4-Scout/Maveee
(Al 2025a).
lenin et al. 2025). MiniMax- 01 (Li et all 2025). Gemini-2.0 wl-.faa/Geeeiniiiii Gethhi-mmiii.... 2.5-fai-2 (DeepMind
2025), GPT mimi-i (Achiam al. 2023
GPT.411 1-mminnni/nanno (OpenAl 2025 and Amazon-Nova
22

---

Published as conference paper at ICLR 2026
Lite (Amazon, 2025). One thing to note is that models in the Gemini series accept at most 10 images as ipput. Therefore, for this series, we use satellite image and 9 street view images per region to stay within the model'ss input constraints.
A.7.2 METRICS
For evaluation, adopt two commonly used metrics in socioeconomic prediction tasks: coefficient of determination (R²) and normalized root mean squared error (nRMSE), Higher R inincicates better performance, with 1.0 representing perfect prediction. Lower nRMSE values indicate more accurate predictions.
A.7.3 CHOICE OF TASK INPUT
While prior work such as Fan et al. (2023) uses 20 or more street view images to represent each region, we find that this setup is often impractical for LVLMs. Specifically, we initially experiment with 20 street view images per region, but observe that this would significantly increase the computational cost, and exceed the input limits models like Gemini, which can only process up to 10 images per inference, and also frequently hit the token length limit of other models. Therefore, adopt a compromise of 10 images per region to ensure compatibility across models while maintaining sufficient visual context.
A.7.4 THE EXAMPLE OF COT PROMPTING
Following the designs of Zhang et al. (2025b) and Xu et 2024), we implement a Chain-of- Thought (CoT)) prompting strategy tailored to the urban soctoeconomic sensing context. Here, we present an example CoT prompt designed specifically for the House Price task.
Suppose you professional real estate appraisal expert Leeds, United Kingdom. Based the provided satellite imagery and several street view photos, please estimate 'house price' in the msoa area where the images taken. Consider factors such as location, visible property features, neighborhood condition, and any other relevant details.
Satellitee Image: <image>
Street View Images: <image> <image> <image> <image> <image> <image> <image> <image> <image> <image>
To perform better on this task, please answer by adopting step-by-step reasoning approach:
Step1 <Summarys:
Explain your overall strategy for estimating the house price based on the given satellite and street view images. Describe how you will approach the task using visual evidence, and mention types of features you plan to focus on.
Step2 <Caption>:
Next, analyze each of the provided street view and satellite images, Clearly list the specific visual features you observe socioeconomic profile
that might affect
prices, summarize these into
Step 3 <Calculation>:
Based on your previous analysis, determine the approximate house-price level for this area and briefly explain the
primary reasons behind your choice.
Step4 <Answer>:
Output single number only representing estimated average house price for the format of
Ennser-NUMBBR<iannww
Figure 13: CoT prompt example for the House Price task.
A.8 ALTERNATIVE OUTPUT DESIGNS AND EVALUATION STRATEGIES
A.8.1 EXPLORING SALTIPLE-CHOIEE STYLE ANSWERING FORMATS
All three evaluation formats used in CityLens be naturally framed as regression tasks. This design choice aligns with established paradigms in recent related works Manvi et al. (2024b), Manvi et al. (2024a) and Zhang al. (2025b), where urban socioeconomic indicators are typically formulated as continuous variables, and numerical prediction remains the primary objective.
We acknowledge that multiple-choice formats can be useful for probing certain model capabilities, such as semantic understanding or categorical reasoning. During the development of CityLens, we explore this possibility by designing multiple-choice version of the Population prediction task. To generate negative choices, we adopt a simple yet controlled strategy: for each ground-trutt population value, three distractors sampled from fixed pool of plausible but incorrect values-fo example,
23

---

Published as a conference paper at ICLR 2026
0.05, 20, and 300. Below is example tested using three models. Preliminary results suggest that while the multiple-choice setting reduces the output space and improves answer interpretability, it also significantly lowers task difficulty compared to free-form regression. Consequently, ultimately opted to focus on open-ended, regression- based evaluation, which we believe more accurately reflects the complexity and realism of urban socioeconomic sensing.
This decision is supported by emerging literature LLM evaluation. Prior studies Aidar Myrza- khan (2024) Li et al. (2024a) have identified several systematic limitations in multiple-choice-base evaluations, including selection bias, position sensitivity, and tendency toward random guess- ing especially in smaller models. These issues may lead to inflated estimates of model capability and fail to capture the inherent complexity the task. Moreover, the multi-choice format may fail to capture the nuanced reasoning or visual understanding required for tasks like urban socioeconomic sensing. In contrast, free-form numerical prediction-despee being more challenging- -ffeers more direct and faithful reflection of model's ability process multimodal information and generate semantically grounded outputs.
A.8.2 STREET VIEW CAPTION EMBEDDING FOR SOCIOECONOMIC REGRESSION
We conduct exploratory experiment inspired by prompt design strategies from Urban VL et al. (2025), applying them to generate captions for street view images. As illustrated in Figure 14a each image is accompanied by contextual information including the city name, geographic coordinates, and scores across 13 predefined visual features. This multimodal input is then fed into a large vision-language model to produce descriptive caption of the scene.
To leverage the semantic richness of these captions, we pass the generated texts through BERT encoder to obtain fixed-length embeddings. These embeddings are subsequently used as input features for downstream regression tasks targeting urban socioeconomic indicators. As shown in Figure T4b this cuption-based embedding approach achieves the best performance the population prediction task, outperforming all other methods.
Method
Population R²
Q: Analyze the image of streetview in (city) in a comprehensive and detailed manner: The coordinate of the streetview image (Longitude). (Latitude). The visual feature score of the streetview image is Facade: (score). Road: (score). Greenery- Shrubs: (score). Creenery-Tree: (score). Street Fumiture: (score).
Direct
-055
&
Normalized
-0.1570
Person: (score). Bike: (score). Heavy Vehicle: (score). Light
Feature-Based
0.2518
Vehicle: (score). Window & Opening: (score). Sidewalk: (score).
Sky: (score). Nature: (score).
Caption-Embed
0.3498
(a)
(b)
Figure 14: (a) Prompt for Caption-Embed Method. (b) Population R² values for different methods.
A.9 BIAS AUDITS
In CityLens, one notable issue is the underrepresentation of Global South cities in some tasks, which may pose risk of reinforeing biases model predictions. To clarify, for several tasks-such as Public Transport Ratio and Mental Tealth-Global South cities are excluded due to the unavailability of high-quality ground-trutt -tunh indicators in these regions.
Building on the analysis presented in Section 3.3 we conduct a preliminary bias audit on the GDP prediction task. Specifically, Wikipedia to categorize cities as follows:
the Global North and Global South classification provided by
Global North: San Francisco, New York, Tokyo, London, Paris, Sydney
Global South: Beijing, Shanghai, Mumbai, Moscow]] Sao Paulo, Nairobi, Cape Town
We then compare model performance the GDP prediction task between these two groups. As shown in the table below, we observe that models perform significantly better on Global North cities,
Note: the classification of Moscow is dehated in some literature.
24

---

Published as a conference paper at ICLR 2026
achieving substantially higher R² and lower nRMSE, suggesting better predictive reliability and fit. This performance gap highlights a geographic disparity in model behavior and may point to underlying biases in how current LVLMs generalize across different socioeconomic and cultural contexts. The fact that these differences emerge under a consistent task formulation and input structure strengthens the case for further bias auditing and fairness-aware model evaluation. Additionally, in Figure 5(a), we observe that Mumbai and Moscow yield significantly negative R² values, further reinforcing the presence of geographic bias in model performance.
Table R2 and nRMSE for Global North and Global South Cities
City
R²
nRMSE
Global North
0.399
0.787
Global South
0.168
0.869
A.10 HYPOTHESES ON LVLMS FOR URBAN SOCIOECONOMIC SENSING
We propose to view visual concepts along two dimensions: low-level and high-level.
Low-level concepts refer to basic, concrete features such greenery, Street Furniture, or vehicle density, which are often directly observable and recognizable by traditional machine learning models.
In contrast, high-level concepts such as signs of poverty, infrastructure quality, or commer- cial activity are abstract constructs that emerge from combinations of multiple visual and contextual signals. These are typically more entangled with socioeconomic meaning and are closer to human-level interpretation.
One of the key strengths of LVLMs lies in their ability to go beyond recognizing low-level features and implicitly perceive and reason about high-level visual semanties. While this ability represents a significant advancement, it also introduces major challenges becomes extremely difficult to analyze whether and how the model correctly interprets high-level visual features. For instance, when it comes to abstract indicators like Bachelor Ratio, it is inherently difficult to isolate single visual factor as the dominant predictor. These outcomes are typically influenced by acombination of visual signals (e.g., signs of poverty, commercial activity, infrastructure conditions), and the relationship between them is complex and entangled.
While these models exhibit impressive capabilities in perceiving a broader spectrum of visual concepts, they also face well-known limitations, most notably hallucination and instability in perception. When identifying low-level features, LVLMs may overlook relevant signals or generate hallucinated content. At the high-level, models may fail to correctly identify interpret key contextual cues related the target indicator, leading to incomplete or biased reasoning, For example, we observe in the BR task that GLM-4.Iv-Thinkin v-TTinkiig tends to over-reason, gradually drifting away from task-relevant semantics.
The above discussion is grounded in our hypothesis about how LVLMMs operate in urban socioeco- nomic sensing tasks: When presented with socioeconomic prediction task, an LVLM may follow a multi-level reasoning process. It first identifies high-level visual concepts that are semantically associated with the target indicator. For example, to estimate the bachelor ratio, the model may consider abstract cues such as signs of poverty, commercial activity, and various other concepts. To support the recognition of these high-level visual concepts, the model must detect corresponding low-level visual features from the input images-such as building density, greenery coverage, and so on. These low-level features are more conerete and directly extractable from satellite or street view images. The model then aggregates these features into high-level visual concepts, which are further cross-checked and composed through reasoning to produce the final prediction.
However, key low-level features are missed or hallucinated during recognition, the resulting high- level concept understanding may be distorted, ultimately leading to inaccurate predictions. Therefore, we believe that understanding why LVLMs struggle with urban socioeconomic sensing is a highly meaningful and non-trivial challenge that deserves deeper investigation. Moreover, the validity of our proposed hypothesis regarding the multi-level visual reasoning process in LVLMs is itself an open question.
25


Reviews:
[{"review_id": "fioE350a6v", "reviewer": "Reviewer_DkC7", "summary": "This paper presents CityLens—a comprehensive benchmark designed to evaluate the ability of Large Vision-Language Models (LVLMs) to predict socioeconomic indicators from satellite and street-view imagery. The benchmark covers 17 cities worldwide, spans six key domains—economy, education, crime, transportation, health, and environment—and defines 11 prediction tasks, assessed under three evaluation paradigms: Direct Metric Prediction, Normalized Metric Estimation, and Feature-Based Regression. The study systematically evaluates 17 state-of-the-art LVLMs, making CityLens the most extensive benchmark to date in terms of geographic coverage, indicator diversity, and model scale. Results show that while LVLMs demonstrate promising perceptual and reasoning capabilities, they still face significant limitations in accurately predicting real-world socioeconomic indicators. CityLens provides a unified framework for diagnosing these challenges and advancing future research in urban intelligence and multimodal socioeconomic analysis.", "questions": "1. I am quite puzzled about the basic motivation of this paper: why choose to predict socioeconomic indicators by feeding street-view or satellite images into large vision–language models (LVLMs)? The existing literature already offers numerous methods specifically designed for modeling urban regions (e.g., region representation learning). I suspect such specialized models may substantially outperform general-purpose multimodal large models in predictive accuracy. Could the authors further clarify the theoretical or practical rationale for choosing LVLMs over traditional urban computing models?\n\n2. I find it difficult to understand how an LVLM can effectively infer complex socioeconomic indicators solely from street-view images. LVLMs were originally designed for semantic understanding of image content (e.g., object recognition, scene description), and their training objectives are not explicitly tied to socioeconomic variables. Thus, establishing a reliable mapping from visual features to abstract socioeconomic indicators seems to lack sufficient mechanistic support. Could the authors explain by what internal mechanisms (e.g., implicit knowledge, vision–language alignment, etc.) the LVLM accomplishes this cross-modal reasoning?\n\n3. Street-view imagery may suffer from serious representational limitations. For example, two areas with vastly different levels of economic development may exhibit highly similar architectural styles or street layouts, causing the model to output similar predictions despite substantial differences in the ground-truth indicators; conversely, areas with similar socioeconomic conditions may yield inconsistent predictions if the street-view sampling locations differ (e.g., arterial roads vs. back alleys). Would the noise and bias introduced by the randomness of image sampling severely undermine predictive reliability? Is reliance on street-view images alone sufficient to support robust socioeconomic sensing?\n\n4. In the “Region” column of Figure 3b, the label “sat” appears. Does this denote satellite imagery? If so, how do the authors extract socioeconomic information from region-level satellite images? Compared with street views, what complementary or more critical visual cues do satellite images provide?\n\n5. I am very interested in the results of Figures 5b and 5c. According to recent work such as FlexiReg: Flexible Urban Region Representation Learning, satellite imagery is often more discriminative than street-view imagery for urban representation. Do the results in this paper support that conclusion as well? Could you further explain why satellite imagery might be more effective than street-view imagery for predicting socioeconomic indicators? What differences in spatial scale or semantic information underlie this discrepancy?\n\n6. I consider the “Prompt Design and Case Analysis” section to be one of the most insightful parts of the paper, as it directly reveals how the LVLM is guided to perform structured scoring. Given the methodological importance of these details, I recommend moving this content into the main text rather than the appendix, so that readers can better understand the coupling between model behavior and task design.\n\n7. I am intrigued by the ablation studies on vision foundation models such as DINO and SigLIP presented at the end of the paper. Could the authors elaborate on the functional differences between these pure vision encoders and end-to-end multimodal large models (e.g., LLaVA, Qwen-VL) for this task? Which class of models contributes more to the final predictive performance? Is it reasonable to conclude that high-quality visual representations are the key to success, while the language model primarily serves as a “scoring interface”?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. CityLens introduces the most extensive benchmark to date for evaluating LVLMs on urban socioeconomic prediction, with unprecedented coverage across 17 globally distributed cities, 11 diverse tasks, and 6 critical socioeconomic domains. This scale and diversity significantly advance beyond prior urban vision or geospatial AI benchmarks.\n\n2. Beyond reporting performance numbers, the paper offers thoughtful discussion on why LVLMs struggle—such as lack of numerical grounding, sensitivity to visual ambiguity, and collapse to city-level averages—thereby guiding future model design and training strategies for urban perception tasks.", "weaknesses": "Please refer to questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4PAt2yp8jK", "reviewer": "Reviewer_pPnJ", "summary": "The paper presents CityLens, a comprehensive benchmark for evaluating Large Vision-Language Models (LVLMs) on urban socioeconomic sensing using satellite and street-view imagery. Covering 17 cities, 11 indicators, and six domains, CityLens defines three evaluation paradigms and benchmarks 17 LVLMs. Results show that while LVLMs capture some visually grounded socioeconomic patterns, they perform poorly on abstract or weakly visual indicators, highlighting key limitations and opportunities for future research in multimodal urban analytics.", "questions": "Has the author considered including domain-specific models such as UrbanVLP, UrbanCLIP, or UrbanGPT, as well as other fine-tuned LVLMs, as part of the baselines? These models incorporate spatial alignment or social semantic injection mechanisms during training, which may better align with the research objectives of this paper. Analyzing how their training paradigms affect the results would provide deeper insights and elevate the work from a mere model evaluation study to a training paradigm analysis at the methodological level.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes CityLens, a large-scale and multi-domain benchmark designed to evaluate the performance of LVLMs on urban socioeconomic indicator prediction tasks. It represents a pioneering attempt in this research area.\n2. The dataset covers 17 cities, 11 socioeconomic indicators, and 17 LVLMs, with a large experimental scale and carefully designed data collection, mapping, and preprocessing pipelines. The paper introduces three distinct evaluation paradigms—Direct Metric Prediction, Normalized Estimation, and Feature-Based Regression—providing a multi-perspective analytical framework. Both data and code are publicly available, ensuring strong reproducibility and potential academic impact.\n3. Beyond comparing model performance, the paper also investigates the effects of geographic variation, input modality (satellite vs. street view), the number of visual features, and Chain-of-Thought (CoT) reasoning strategies.", "weaknesses": "1. The results in the Direct Metric Prediction and Normalized Estimation sections (with most tasks showing R² below 0.2 or even negative) clearly demonstrate the severe limitations of current LVLMs in numerical prediction tasks. From a methodological perspective, relying on prompting to have LVLMs directly output numerical values may be inherently unstable. Such a mechanism is fundamentally ill-suited for precise regression tasks.\n2. The current experiments primarily focus on comparing multiple general-purpose LVLMs (e.g., GPT-4o, Gemini, Qwen2.5VL), which are not specifically optimized for urban or geospatial applications. As a result, the findings mainly reflect the deficiencies of generic LVLMs on urban tasks rather than revealing the key factors that could improve model performance.\n3. Although the paper presents extensive experimental results, it lacks deeper mechanistic analysis explaining why models perform poorly and which visual or linguistic factors contribute to these errors. For instance, indicators with strong visual correlations such as Building Height and GDP perform relatively well, whereas tasks like Life Expectancy and Mental Health perform extremely poorly—an observation that deserves more in-depth discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GyMjpkGlVY", "reviewer": "Reviewer_gfN8", "summary": "This paper proposes CityLens, a comprehensive benchmark dataset designed to evaluate the capabilities of Large Vision-Language Models (LVLMs) in predicting urban socioeconomic indicators from satellite and street view imagery. Specifically, this multimodal dataset covers 17 cities across 6 continents, 6 socioeconomic domains (economy, education, crime, transport, health, environment), and 11 specific prediction tasks. Building upon this, the authors systematically evaluate 17 state-of-the-art LVLMs using three distinct paradigms: Direct Metric Prediction, Normalized Metric Estimation, and Feature-Based Regression. The key findings reveal that while LVLMs show promise, they generally struggle with accurate and generalizable socioeconomic prediction, with performance varying significantly across tasks, models, and geographic locations.", "questions": "1. In the Feature-Based Regression paradigm, the authors show that CLIP-based encoders perform best. Was the satellite image used as an input to the LVLM during this feature extraction step, or was this based purely on the street view images? Could the superior performance of CLIP be attributed to its training on web-scale image-text pairs, which might include more diverse urban scenes?\n2. Did you try nonlinear regressors for the 13-attribute vectors (RF/GBM/MLP)? If not, is the conclusion “LVLM features aren’t sufficient” or “the linear readout is underpowered”?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The curated CityLens dataset is a major contribution to the community.\n2. The benchmarking results are comprehensive and have a large coverage of state-of-the-art LVLMs.\n3. The use of three evaluation paradigms (Direct, Normalized, Feature-Based) is technically sound.\n4. The paper is well organized and easy to follow.", "weaknesses": "1. The evaluation is primarily zero-shot or few-shot. A natural question is how much these models could improve if fine-tuned on the CityLens dataset. While the authors mention this as a future direction, a preliminary fine-tuning experiment on one or two models would have strengthened the paper by establishing a potential performance upper bound.\n2. While the paper diagnoses what models struggle with (e.g., mental health, life expectancy), a more detailed qualitative analysis of why could be beneficial. For instance, providing examples of specific visual cues that models misinterpret or fail to utilize for the most challenging tasks would offer deeper mechanistic insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wB8iQ9Pvnn", "reviewer": "Reviewer_Zxk9", "summary": "This paper introduces CityLens, which is a benchmark for evaluating the capabilities of LVLM in urban socioeconomic deduction. The benchmark is comprehensive acorss 11 prediction tasks across 6 domains e.g., economy and health. The authors evaluate 17 SOTA LVLMs using three evaluation protocols. The primary findings are that current LVLMs struggle significantly with these tasks and that the feature regression method outperforms the direct and zero-shot predictions.", "questions": "What would happen if you only use 1 satellite image and compare it with 10 street ones?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "The primary strength of this paper is the creation of a large-scale dataset for this urban tasks. The effort in collecting, aligning, and processing satellite and street view images from multiple sources, and 11 distinct socioeconomic indicators is non-trivial. The dataset could be of use for other research efforts.\nThe reproducibility is very good. The authors provide an alternative version of the dataset using only publicly accessible Mapillary images (CityLens-Mapillary), which is a commendable step towards transparency and reproducibility.", "weaknesses": "1.This paper's central contribution is a new benchmark rather than a new method. The models and evaluation techniques are nothing new. This is not in line with the scope of ICLR\n\n2The main takeaway is that current LVLMs perform not good on this complex task with many R2 scores near 0. I admit \"negative results\" are valuable, the paper offers limited deep insight into the reasons behindwhy these representations fail.\n\n3The finding that COT is task-dependent is already a well-established consensus in the broader LLM community. The claim that specific vision encoders CLIP outperform others is also expected.\n\n4.The comparison is not fair. Feature based methods have a LASSO regressor while other evaluation protocols do not have.\n\n5.The finding that satellite imagery has \"minimal impact\" is based on an unbalanced comparison of 1 satellite image vs. 10 street view images. It is unsurprising that the information from 10 ground-level images dominates the information from one overhead image.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)