CityBench: Evaluating the Capabilities of Large Language Models for Urban Tasks

## Abstract

Abstract Large language models (LLMs) with powerful generalization ability has been widely used in many domains. A systematic and reliable evaluation of LLMs is a crucial step in their development and applications, especially for specific professional fields. In the urban domain, there have been some early explorations about the usability of LLMs, but a systematic and scalable evaluation benchmark is still lacking. The challenge in constructing a systematic evaluation benchmark for the urban domain lies in the diversity of data and scenarios, as well as the complex and dynamic nature of cities. In this paper, we propose CityBench , an interactive simulator based evaluation platform, as the first systematic evaluation benchmark for the capability of LLMs for urban domain. First, we build CitySim to integrate the multi-source data and simulate fine-grained urban dynamics. Based on CitySim , we design 7 tasks in 2 categories of perception-understanding and decision-making group to evaluate the capability of LLMs as city-scale world model for urban domain. Due to the flexibility and ease-of-use of CitySim , our evaluation platform CityBench can be easily extended to any city in the world. We evaluate 13 well-known LLMs including open source LLMs and commercial LLMs in 13 cities around the world. Extensive experiments demonstrate the scalability and effectiveness of proposed CityBench and shed lights for the future development of LLMs in urban domain. The dataset, benchmark and source codes are openly accessible to the research community via https://github.com/tsinghua-fib-lab/CityBench .

## 1 Introduction

Recent years, large language models (LLMs) with strong generalization and reasoning capabilities have achieved excellent results in various fields  [, ], e.g., programming [], mathematics [], VQA [], common sense reasoning [, ] and so on. Furthermore, powerful LLMs enable many unimaginable research endeavors to become feasible, e.g., agent [], embodied intelligence [] and AGI []. These researchers postulate that LLMs, by acquiring extensive world knowledge and common sense, hold the key to unlocking promising outcomes in these challenging applications. Many works [, , ] have demonstrated that LLMs do indeed possess a wealth of world knowledge and can be regarded as world models to some extent, while other works [, ] indicate that LLMs are not real world models and they lack an comprehensive understanding of the real world. However, these research efforts have primarily focused on small-scale world modelling, such as those works conducted in virtual-home [], while overlooking the large-scale modeling in urban space [, ].

Various works have explored the potential of LLMs in modeling urban space and solving urban tasks. For example, researchers evaluate the potential of LLMs on remote sensing understanding tasks [, ] and urban visual tasks [, ]. Gurnee et al. [] and Roberts et al. [] evaluate whether LLMs acquire the spatial knowledge of the world, such as cities and coordinates. Manvi et al. [, ] try to extract the geospatial knowledge in LLMs to conduct geospatial indicator prediction tasks []. Besides, researchers also explore how to apply LLMs into the realistic urban applications, e.g., traffic control [, ], traffic prediction [], mobility prediction [], visual language navigation [] and so on. However, on the one hand, these existing works primarily focus on evaluating the static spatial knowledge of LLMs without considering the environment dynamics and interactivity, which are the key point for the world model []. On the other hand, most of them only focus on one type of task and one modality of data in the urban space, using small dataset that are not scalable globally. The root cause of their shortcomings is the insufficient integration of multi-source data for urban spaces and the lack of suitable interactive city simulators. Although there are some existing simulators for urban space such as game simulators [] and traffic simulators [], they are either not open-source, or tailored to specific problems. All of them do not support systematically evaluating the capability of LLMs as world models.

Figure: Figure 1: The framework of global evaluation benchmark CityBench, which consists of a simulator CitySim and 7 representative urban tasks. We can select any city around the world to automatically build new benchmark for it.
Refer to caption: x1.png

In this paper, we propose CityBench, a comprehensive evaluation platform for assessing the capability of LLMs as city-scale world models. It covers multiple modalities, supports interactive simulations, and is easily extensible globally. CityBench consists of two modules: a simulation module CitySim for integrating multi-source urban data and simulating urban dynamics, a evaluation module Benchmark for various evaluation of LLMs. In CitySim, we first collect three kinds of open source urban data: geospaital data from Open Street Map, urban vision data including from Google Map, and human activity data from Foursquare and other websites. Then, we build an efficient GPU based engine to simulate individual behaviors in the urban environment and develop various interfaces for controlling the urban dynamics and sensing the urban environments. Furthermore, based on CitySim, we design a comprehensive benchmark to evaluate the capability of LLMs as city-scale world models, covering core research problems from various urban research fields. The evaluation benchmark comprises two levels of tasks: geospatial understanding tasks and decision-making tasks. In geospatial-understanding tasks, based on the integrated multi-source data from CitySim, we introduce street view&satellite image understanding and urban space knowledge understanding tasks to evaluate the basic capability of LLMs as city-scale world models. In decision-making tasks, we apply LLMs to interact with CitySim to complete the mobility prediction task, traffic signal control task and street navigation task which require comprehensive ability of LLMs as city-scale world models. In summary, our contribution are as follows,

- •
We build CitySim, an efficient simulator for integrating multi-source urban data and simulating find-grained individual behaviors in the urban environments. It provides ease-of-use APIs for controlling urban dynamics and sensing urban environments.
- •
We propose CityBench, a comprehensive evaluation benchmark for evaluating the capability of LLMs as city-scale world models, which includes 4 multi-modal geospatial understanding tasks and 3 interactive urban decision-making tasks.
- •
Extensive experiments on CityBench with 13 well-known open source and commercial LLMs demonstrate the effectiveness of CityBench as evaluation benchmark and also present the limitation of applying LLMs as world models in urban tasks.

## 2 Related Work

LLM Evaluation for Urban Space
Researchers from various urban related fields have conduct extensive evaluations of LLM in urban space from different aspects. Zhang et al [] and Kuckreja et al. [] evaluate the performance of multi-modal LLMs on several remote sensing related tasks. Yang et al. [] propose V-IRL benchmark to evaluate the performance of multi-modal LLMs on street view image related tasks including localization and recognition tasks. Mai et al. [] and Manvi et al. [] use LLMs to predict social indicators like population and education level. Gurnee et al. [] and Bhandari et al. [] try to testify whether LLMs know the coordinates of geospatial entity. Mooney et al. [] and Deng et al. [] use GIS exams to understand the gospatial skills of LLMs. Roberts et al. [] design GPT4GEO to evaluate the geospatial capacity of LLM with limited case studies. Different from these works with single aspect, our work covers most of the above problems in one benchmark with different modalities and views. Furthermore, we also support interactive simulation based various decision making task evaluation which enable more realistic and challenging performance evaluation.

Interactive Decision Making and Urban Simulator
Beyond the above static evaluation, researchers also evaluate the capacity of LLMs in the interactive decision making tasks with customized simulators, e.g., web agent [, ] with web environment and embodied intelligence [] with virtual home []. In the urban domain, Schumann et al. [] apply LLM to do the visual language navigation task in Touchdown [] and Lai [] apply LLMs as the traffic light controller in CityFlow [] to manage the road traffic. Besides, Yang et al. [] design V-IRL as the environment of street view image related tasks and propose a global scale virtual intelligence benchmark. These works only evaluate the potential of LLMs in single urban decision making task and most of their results rely on small-scale datasets in limited regions. Different from them, our work builds on an efficient urban simulator with global scale and supports three representative urban decision making tasks with different modality in one benchmark, including mobility prediction [], traffic control [] and street navigation. In the future, it will support more kinds of urban tasks, e.g., urban planning [].

## 3 CitySim

In the section, we present an overview of the simulation module CitySim.
The module first integrates multi-source urban data to support multi-modal urban sensing and urban dynamic simulation.
Next, in order to reconstruct real urban dynamics and individual behaviors, the module also provides a efficient GPU-based simulation engine with ease-to-use sensing and controlling APIs.

### 3.1 Datasets

To present a complete picture of the city’s geospatial structure, semantic features, and human activities, CitySim integrates the following globally available data from multiple sources.

Geospatial Data
Geospatial data, represented by maps, is the most fundamental data for describing the urban structure including buildings, road networks, green areas, etc.
OpenStreetMap (OSM) (^1^11[https://openstreetmap.org/](https://openstreetmap.org/)) is an open source map data provider that provides researchers with globally available geospatial data.
However, the raw data provided by OSM cannot support the simulation of urban dynamics directly, and the relationship between different elements is incomplete such as the connection between buildings and roads.
Therefore, we provide a globally available map building tool (^2^22[https://github.com/tsinghua-fib-lab/mosstool](https://github.com/tsinghua-fib-lab/mosstool)) that reconstructs lanes, lane topology, and building-lane connections based on the raw OSM data.
The reconstructed map is used as the geospatial base and simulation input in CitySim.
Based on the map, APIs such as search by ID, search by relationship, and search by spatial range are provided in the CitySim module to help LLMs access the city geospatial information.

Urban Visual Data
Street view data and satellite images are two types of urban data that contains rich semantic information, which can be used to evaluate the capacity of multi-model LLMs.
In addition, the two types of data are also publicly available globally.
Therefore, CitySim also integrates the two types of data, the former obtained via Google Maps API and Baidu Maps API, and the latter using the Esri World Imagery(^3^33[https://www.arcgis.com/home/item.html?id=10df2279f9684e4a9f6a7f08febac2a9](https://www.arcgis.com/home/item.html?id=10df2279f9684e4a9f6a7f08febac2a9)) as data source.
In the CitySim module, street view data is accessed through spatial location and facing direction, and satellite images are acquired through spatial ranges.

Human Activities Data
We use the open-source global Foursquare-checkin [] data and a synthetic global origin-destination data (OD data) [] as the proxy of human activities to enable the fine-grained human movement simulation. The Foursquare-checkin [] dataset is a long-term user checkin data collected from Foursquare (^4^44[https://foursquare.com/](https://foursquare.com/)) around 415 cities in the world. Origin-destination data is generated by a diffusion model with population from Worldpop(^5^55[https://www.worldpop.org/](https://www.worldpop.org/)) and satellite image from Esri World Imagery as input. While all the user information are anonymized, we follow the license from Foursquare-checkin [] to protect the public privacy.

### 3.2 Simulation

Cities have strong dynamics triggered by human activities, such as traffic flow.
Such dynamics are a strong challenge to the capacity of LLMs.
To simulate urban dynamics, we implement GPU-accelerated individual-level simulations of human activities.
The simulator takes the map reconstructed from OSM and the travel demand described by the synthetic OD data, and simulates the mobility behaviors through individual motion models such as the car-following model [] and the lane-change model [] to obtain the information about the dynamics of all pedestrians and vehicles in the city every second.
The simulator also provides a series of sensing and control APIs.
Through the sensing APIs, LLM can obtain data about urban dynamics such as pedestrian location, vehicle speed, and road average speed.
Through the control API, LLM can intervene in the city’s operation, such as modifying traffic signal lights, modifying the speed limit of the road, etc.

Overall, with the CitySim module, we provide globally available integrated multi-source data with an interactive urban dynamic simulator to multi-model LLMs.

## 4 CityBench

Based on CitySim, we introduce a global multi-modal urban evaluation benchmark CityBench to evaluate the capability of multi-modal LLMs as the city-scale world models with 7 urban tasks. These tasks are 4 perception and understanding tasks and 3 decision making tasks, detailed design of these tasks are introduced as follows.

### 4.1 Perception and Understanding

In this section, we introduce the perception and understanding task group. The first task is the street view image geolocalization task from the urban visual intelligence []. Following are social indicator prediction and infrastructure inference tasks from remote sensing field. Finally, we adapt GeoQA [] task into urban environment.

#### 4.1.1 Image Geolocalization

Image geolocalization task is to predict the precise location of image based its context. Street view image is regarded as the recording of urban appearance and play an important role in understanding the urban environment and dynamics []. Thus, we query multi-modal LLM with street view image and require it to directly generate the city name of its location and precise latitude and longitude coordinates. A good multi-modal LLM should recognize the important objects from the street image and mapping them into the potential locations. Following [], we define two metrics: city inference accuracy and accuracy within 1km/25km of the correct location to measure the performance.

#### 4.1.2 Geospatial Prediction

Geospatial predictions are important for understanding the global sustainable development especially for developing countries, e.g., poverty estimation [] and population density estimation []. One of the most widely used solutions is using satellite images with machine learning methods to predict these socioeconomic indicators. In the benchmark, following the setting from [], we query multi-modal LLM with a satellite image as context to predict the population density of it. We use population dataset from Worldpop [] as the groundtruth and Pearson’s $r^{2}$ and RMSE as metrics.

#### 4.1.3 Infrastructure Inference

Besides, we also introduce the infrastructure inference task which means to recognize the urban infrastructures from the satellite images. This task require the ability of scene understand and object segmentation of urban environment. The groundtruth of urban infrastructure of each satellite image is extracted from the OSM by matching predefined infrastructure key words within a fixed spatial range. Given the satellite image and the list of all kinds of infrastructures, multi-modal LLM is required to generate the infrastructure names appeared in the image. In the benchmark, we pay attention to the following infrastructures: Airport, Harbor, Stadium, Bridge, Roundabout and Train Station.

#### 4.1.4 GeoQA for City Elements

Beyond understanding the urban space from the visual perspective, we introduce geographic question answer(GeoQA) [] to test whether LLMs comprehends the fundamental spatial elements [] in a city from the concept view, such as road, landmarks and boundary. For example, we directly ask LLM about the relation between different roads in a city. Following [], we classify the spatial elements into six groups and design several single choice problems for each group. These six groups are node, path, landmark, boundary, districts and others.

### 4.2 Planning and Decision Making

Different from the static evaluation introduced in the last section, we design three interactive decision making tasks to evaluate the capability of LLMs in dynamic and partial observed environments which are more challenging and realistic. With the interaction with the CitySim simulator and dynamic human activities, LLM needs to understand the important mechanisms and regularity in the urban environments to complete the decision marking tasks. Detailed design of decision making tasks are introduced as follows.

#### 4.2.1 Mobility Prediction

Mobility prediction task is one of the fundamental task for understanding the human behaviors and urban dynamics. Mobility prediction task is to predict the next location of user in the next time window with given the past mobility trajectory. Here we use the the global foursquare checkin data as the proxy of human mobility and integrate it into the simulator to support the mobility prediction task. We follow [] to conduct the mobility prediction task via LLMs.

#### 4.2.2 Street Navigation

Street navigation task is widely used in neurocognitive science [] and visual language navigation [] as the important benchmark for evaluating the spatial cognition of human and models. Here, we also define a text based street navigation task to evaluate the zero-shot navigation capability of LLMs in a new city. Different from the visual language navigation which require model to follow the language instruction and understand the scene via street view image, our street navigation task require model to explore the region via the local information (e.g. accessed road names) provided by the simulator during action and its intrinsic knowledge of the whole urban environment.

#### 4.2.3 Traffic Signal Control

Traffic signal control task is one of the widely studied realistic urban decision making task in recent years []. It is challenging for existing methods due to the dynamic traffics and the generalization issues. Traffic signal control task is to generate the future traffic signal schedule by considering the current traffic states and the future traffics. Lai et al. [] propose LLMLight to solve the single traffic signal control in simple intersection and demonstate the generalization of LLMs. Follow this work, we extend it into multiple traffic signal control in a region with more than one intersections, which are more challenging and require cooperation between different intersections.

## 5 Benchmark and Experiments

### 5.1 Settings

Dataset
We collect multi-source data for 13 cities around the world, including street view& satellite image, geospatial data and human activity data. Detailed information about these dataset are presented in Table 1. Our platform can expand to any city in the world with just inputting its name. The effortless evaluation across various regions and cultures makes it a effective world model evaluator of LLMs. CityBench complies with the Google Maps Platform license(^6^66[https://cloud.google.com/maps-platform/terms](https://cloud.google.com/maps-platform/terms)), the Baidu Maps Platform license(^7^77[https://map.baidu.com/zt/client/service/index.html](https://map.baidu.com/zt/client/service/index.html)) and the Esri Master License Agreement(^8^88[https://www.esri.com/en-us/legal/terms/full-master-agreement](https://www.esri.com/en-us/legal/terms/full-master-agreement)).

**Table 1: Multi-source data for 13 cities around the world. For simplifying the processing steps, we have defined these spatial range of collected data within rectangular boundaries. Most cities only include the central part of the city’s administrative boundaries, which may lead to some inaccuracies. Specific coordinate settings can be found in the appendix.**
| World | Cities | Visual Data | GeoSpatial Data | Human Activity Data |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Satellite<br>Image | Street View<br>(sampling) | Roads | PoI/AoIs | OD flow<br>(>10) | Checkins |  |  |
| Asia | Beijing | 1764 | 1000 | 17043 | 276090 | 1905025 | 21015 |
| Shanghai | 5925 | 1000 | 33321 | 57731 | 845188 | 33129 |  |
| Mumbai | 638 | 1000 | 6296 | 60245 | 309147 | 31521 |  |
| Tokyo | 1120 | 1000 | 33174 | 1146094 | 969865 | 1044809 |  |
| Europe | London | 1710 | 1000 | 14418 | 83892 | 1401404 | 173268 |
| Paris | 238 | 1000 | 4443 | 21950 | 28362 | 85679 |  |
| Moscow | 1558 | 1000 | 9850 | 28289 | 979064 | 836313 |  |
| Americas | NewYork | 320 | 1000 | 5414 | 349348 | 71705 | 390934 |
| SanFrancisco | 345 | 1000 | 4171 | 73777 | 61367 | 100249 |  |
| SaoPaulo | 1332 | 1000 | 28714 | 1681735 | 311830 | 808754 |  |
| Africa | Nairobi | 336 | 1000 | 2972 | 264101 | 135332 | 25727 |
| CapeTown | 896 | 1000 | 5947 | 151711 | 525578 | 11591 |  |
| Oceania | Sydney | 1935 | 1000 | 21390 | 141997 | 438763 | 54170 |

Baselines We select state-of-the-art open source LLMs and commercial API to evaluate the performance of LLMs on the benchmark. For multi-modal LLMs, open source LLMs are LLaVa-NeXT [], CogVLM-v2 [], MiniCPM-LLama3-V-2.5 [], commercial APIs are Qwen-VL-plus and GPT4o. For LLMs, open source LLMs are LLama3-8B, LLama3-70B, Mistral-7B-v0.2 [], Mixtral-8x22B-v0.1 [] and DeepSeekv2 [], commercial APIS are GPT3.5, GPT4 []. We test most of the open-source LLMs by using the API from DeepInfra (^9^99[https://deepinfra.com/](https://deepinfra.com/)), while the commercial models using their official APIs (^10^1010[https://platform.openai.com/docs/api-reference/introduction](https://platform.openai.com/docs/api-reference/introduction)). We also deploy several multi-modal LLMs with 1 A100 GPU, including LLaVa-NeXT [], CogVLM-v2 [], MiniCPM-LLama3-V-2.5 [].

Evaluation Metrics
We follow the common practice of each task to define the metrics. Metrics and instances for each task are presented in Table 2. For each task with results from 13 cities, we report the mean value and standard deviation value of them in the following section.

**Table 2: Detailed information of 7 evaluation tasks in CityBench, including data modality, metric and data instances. Task settings across different cities keep consistent.**
| CityBench | Tasks | Modality | Metrics | Instances |
| --- | --- | --- | --- | --- |
| Perception&<br>Understanding | Image Geolocalization | Image | Accuracy, Accuracy@1km/25km | 500 $\times$ 13 cities |
| Geospatial Prediction | Image | $r^{2}$, RMSE | ~500 $\times$ 13 cities |  |
| Infrastructure Inference | Image | Accuracy, Recall | ~500 $\times$ 13 cities |  |
| GeoQA for City | Text | Accuracy | ~1000 $\times$ 13 cities |  |
| Planning&<br>Decision Making | Mobility Prediction | Text | Top1-Acc, F1 | 500 $\times$ 13 cities |
| Street Navigation | Text | Steps, Success Rate | 50 $\times$ 13 cities |  |
| Traffic Signal Control | Text | Travel Time, Queue Length, Throughput | 1 regions $\times$ 3600s $\times$ 13 cities |  |

### 5.2 Main Results

#### 5.2.1 Perception and Understanding

Results of visual perception tasks of multi-modal LLMs are presented in Table 3. In general, commercial LLMs like GPT4o performs best in most tasks, achieving the best results in 5 out of 7 metrics. Among the open source models, LLaVA-NeXT-34B performs best with achieving 3 second best results in 7 metrics. Besides, we can find that the performance of open source LLM varies significantly across different cities which indicates significant geospatial bias of open source LLMs. Detailed analysis about this can refer to section 5.2.3. Results of geospatial knowledge QA task of LLMs are presented in Table 4. As Table 4 shows, commercial LLMs GPT4-Turbo performs best in 4 out of 6 tasks and the left two performs second best. In summary, in terms of geospatial knowledge about urban spaces, the open-source LLMs lag far behind the commercial LLMs.

**Table 3: Results of multi-modal LLMs on three representative visual understanding tasks. For most metrics, higher values are better, except for RMSE. In the table, bold denotes the best results, underline denotes the second best results.**
| Models | Image Geolocalization | Population Density | Infrastructures |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| City<br>Accuracy | Accuracy<br>@1km | Accuracy<br>@25km | $r^{2}$ | RMSE | Precision | Accuracy |  |  |
| Open<br>Source | LLaVA-NeXT-8B | 0.267$\pm$0.322 | 0.0097$\pm$0.0269 | 0.221$\pm$0.337 | -0.764$\pm$0.582 | 3.31$\pm$0.823 | 0.245$\pm$0.301 | 0.796$\pm$0.187 |
| LLaVA-NeXT-34B | 0.501$\pm$0.387 | 0.0123$\pm$0.0328 | 0.408$\pm$0.413 | -0.163$\pm$0.603 | 2.61$\pm$0.643 | 0.262$\pm$0.297 | 0.804$\pm$0.187 |  |
| CogVLM2 | 0.559$\pm$0.333 | 0.0067$\pm$0.0176 | 0.326$\pm$0.335 | -0.301$\pm$0.737 | 2.75$\pm$0.504 | 0.287$\pm$0.320 | 0.726$\pm$0.262 |  |
| MiniCPM-V2.5 | 0.262$\pm$0.267 | 0.0061$\pm$0.0124 | 0.223$\pm$0.270 | -1.054$\pm$0.740 | 3.57$\pm$0.621 | 0.251$\pm$0.323 | 0.806$\pm$0.197 |  |
| Commercial<br>API | Qwen-VL-plus | 0.793$\pm$0.288 | 0.0052$\pm$0.0039 | 0.645$\pm$0.313 | -1.028$\pm$1.667 | 3.14$\pm$1.016 | 0.203$\pm$0.233 | 0.454$\pm$0.170 |
| GPT4o | 0.862$\pm$0.079 | 0.0187$\pm$0.0375 | 0.797$\pm$0.148 | 0.122$\pm$0.309 | 2.32$\pm$0.664 | 0.248$\pm$0.306 | 0.812$\pm$0.195 |  |

**Table 4: Results of different LLMs on GeoQA for city, a geospatial knowledge question answering task for urban space. Here, the higher value is better, bold denotes the best results, underline denotes the second best results.**
|  | Models | Nodes | Landmarks | Paths | Districts | Boundary | Others |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Open<br>Source | LLama3-8B | 0.2322$\pm$0.0150 | 0.2765$\pm$0.0143 | 0.3173$\pm$0.0054 | 0.2627$\pm$0.0163 | 0.3569$\pm$0.0109 | 0.3365$\pm$0.0225 |
| Mistral-7B | 0.2286$\pm$0.0167 | 0.1950$\pm$0.0083 | 0.2054$\pm$0.0179 | 0.1511$\pm$0.0150 | 0.3269$\pm$0.0232 | 0.2688$\pm$0.0186 |  |
| LLama3-70B | 0.2646$\pm$0.0076 | 0.3304$\pm$0.0320 | 0.4048$\pm$0.0334 | 0.2532$\pm$0.0429 | 0.3727$\pm$0.0112 | 0.3485$\pm$0.0185 |  |
| Mixtral-8x22B | 0.2633$\pm$0.0155 | 0.3569$\pm$0.0367 | 0.3652$\pm$0.0201 | 0.2730$\pm$0.0370 | 0.3415$\pm$0.0125 | 0.3239$\pm$0.0241 |  |
| Commercial<br>API | DeepSeekV2 | 0.2904$\pm$0.0065 | 0.3035$\pm$0.0165 | 0.4133$\pm$0.0208 | 0.3518$\pm$0.0399 | 0.4650$\pm$0.0363 | 0.3264$\pm$0.0232 |
| GPT3.5-Turbo | 0.2477$\pm$0.0260 | 0.2627$\pm$0.0128 | 0.3222$\pm$0.0255 | 0.1909$\pm$0.0244 | 0.3788$\pm$0.0137 | 0.3050$\pm$0.0244 |  |
| GPT4-Turbo | 0.3429$\pm$0.0095 | 0.3938$\pm$0.0393 | 0.5249$\pm$0.0349 | 0.3054$\pm$0.0412 | 0.4485$\pm$0.0198 | 0.3707$\pm$0.0219 |  |

#### 5.2.2 Panning and Decision Making

Table 5 presents the results of mobility prediction task and street navigation task, the performance difference between LLMs in decision making tasks are much smaller than the last section. It is amazing to observe that open source LLMs (e.g., LLama3-70B) performs better than commercial LLMs in these two tasks. Results of traffic signal control task are presented in Table 6. Here, we report results from three cities with various traffic volume and environment. In general, commercial LLMs like GPT-3.5 or GPT-4 performs better but small open source LLMs can also achieve competitive results in some cases. One interesting direction is to investigate these difference and find out the underlying factors behind them.

**Table 5: Results of LLMs on mobility prediction and street navigation tasks. The arrow next to the metric indicates the direction of improvement, an upward arrow indicates that higher values of the metric are better.**
| Models | Mobility Prediction | Street Navigation |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Top1<br>Accuracy $\uparrow$ | F1 $\uparrow$ | Success<br>Rate $\uparrow$ | Average<br>Steps $\downarrow$ |  |  |
| Open<br>Source | LLama3-8B | 0.130$\pm$0.0320 | 0.094$\pm$0.030 | 0.747$\pm$0.140 | 5.304$\pm$2.059 |
| Mistral-7B | 0.090$\pm$0.0255 | 0.087$\pm$0.031 | 0.730$\pm$0.134 | 5.382$\pm$1.982 |  |
| LLama3-70B | 0.159$\pm$0.0430 | 0.130$\pm$0.034 | 0.796$\pm$0.085 | 4.941$\pm$1.592 |  |
| Mixtral-8x22B | 0.155$\pm$0.0419 | 0.136$\pm$0.042 | 0.745$\pm$0.133 | 5.339$\pm$1.961 |  |
| Commercial<br>API | DeepSeekV2 | 0.126$\pm$0.0520 | 0.101$\pm$0.044 | 0.698$\pm$0.119 | 5.739$\pm$1.812 |
| GPT3.5-Turbo | 0.152$\pm$0.0388 | 0.113$\pm$0.035 | 0.719$\pm$0.143 | 5.473$\pm$2.015 |  |
| GPT4-Turbo | 0.147$\pm$0.0357 | 0.125$\pm$0.035 | 0.757$\pm$0.091 | 5.184$\pm$1.520 |  |

**Table 6: Results of LLMs on traffic signal tasks on three cities with representative traffic volume. We used real-world mobility demands to generate traffic to support the simulation. The traffic signals we actually control in the experiment are only a part of the trajectory path, while the remaining signals are operated according to the default fixed-time scheduler.**
| Models | Paris | Newyork | Beijing |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Throughput | Travel<br>Time | Queue<br>Length | Throughput | Travel<br>Time | Queue<br>Length | Throughput | Travel<br>Time | Queue<br>Length |  |
| LLama3-8B | 205 | 2936.579 | 652.157 | 580 | 2635.365 | 86.993 | 2128 | 1873.757 | 40.941 |
| Mistral-7B | 183 | 2940.694 | 651.413 | 574 | 2634.054 | 87.807 | 1696 | 2059.84 | 57.759 |
| Llama3-70B | 193 | 2937.528 | 650.855 | 516 | 2664.102 | 89.116 | 2031 | 1893.517 | 43.475 |
| Mixtral-8x22B | 179 | 2938.643 | 653.168 | 526 | 2655.111 | 88.567 | 1604 | 2093.986 | 63.006 |
| DeepSeekv2 | 193 | 2938.567 | 653.532 | 495 | 2682.194 | 89.914 | 2233 | 1754.923 | 32.771 |
| GPT3.5-Turbo | 214 | 2935.621 | 651.704 | 544 | 2649.214 | 87.926 | 2199 | 1778.963 | 34.965 |
| GPT4-Turbo | 191 | 2937.580 | 652.026 | 495 | 2670.238 | 89.706 | 2240 | 1746.608 | 32.234 |

#### 5.2.3 Geospatial Bias Analysis

To further investigate the difference between LLMs, we report the detailed results of mobility prediction task and image geolocalization tasks from 13 cities in Figure 2. Based on the above results, we have made several interesting discoveries. First, we find that the performance of different LLMs varies a lot across different cities, no LLM can always perform best in mobility prediction tasks. Second, we find that the performance of multi-modal LLMs on visual task like image geolocalization task are significantly biased. Most LLMs perform well in major international cities, but poorly in some lesser-known cities (e.g., CapeTown). The variability in evaluation results demonstrate the necessity of establishing a global evaluation benchmark, and also highlights the potential shortcomings and areas for improvement of LLMs.

Figure: Figure 2: Detailed results of LLMs on mobility prediction task(left) for different cities and image geolocalization task(right) for different models. In left figure, the order of cities in each group is decided by the prediction performance of GPT3.5: Tokyo, Paris, London, Mumbai, Moscow, CapeTown, Nairobi, SanFrancisco, Sao Paulo, Shanghai, Sydney, NewYork, Beijing.
Refer to caption: x2.png

## 6 Discussion

Limitations. While our platform is based on the public data from various sources, the quality of different data may play a important role in the evaluation results. For example, The varying levels of map data and street view data quality across different regions worldwide may significantly negatively impact the results. Besides, due to the limitation of global label data for various task, we only select 7 urban tasks in current benchmark which only cover portion of the research problems in urban domain. In the future, we plan to collect more kinds of tasks with global scale groundtruth data to further improve the reliability and representativeness of benchmark.

Ethical considerations and potential societal impact. Our benchmark is designed for enable the global evaluation of LLMs for various cities with different cultures and countries. We try our best to improve the ease-of-use and fairness for cities with different development levels. However, as discussed in section 5.2.3, due to the limitation of accessed data, the evaluation results for different cites varies a lot. Therefore, the variation in evaluation results caused by data quality may lead to a certain degree of misunderstanding regarding the performance on some urban problems. We call the whole community for attention to this issue to improve the usability of LLMs across different races and countries, promoting fairness and sustainable development of the world.

Develop foundation model for urban domain. Based on the results of our benchmark, we find existing LLMs perform poorly on many urban tasks, even worse than some classic simple baseline algorithms. Developing LLMs tailored for urban domain is urgently necessary. We hope our benchmark can accelerate this development and we look forward to a more comprehensive and robust evaluation framework for urban domain.

## 7 Conclusion

In this paper, we propose CityBench, a systematic evaluation benchmark for LLMs in urban domain. In CityBench, we first build CitySim to integrate multi-source urban data and provide the interactive ability of simulating fine-grained urban dynamics. Based on CitySim, we design 7 important urban tasks to constitute the CityBench for evaluating the capabilities of LLMs in the urban domain from various aspects. Results on 13 cities around the world demonstrate the effectiveness and scalability of CityBench. In the future, we plan to further expand CityBench to include a wider range of tasks, thereby facilitating more robust and representative evaluation of LLMs for urban domain research.

## Appendix A Appendix

### A.1 Parameter Settings

In the experiments, the default generation parameters for LLMs are presented in Table 7. Besides, we modify some parameters for some experiments. For street navigation tasks, the temperature is changed to 0.7 for better exploration. For multi-modal LLM, the temperature of CogVLM2 is 0.8 and the temperature of MiniCPM-V-2.5 is 0.7.

**Table 7: Generation parameter settings for LLMs during the experiments.**
| Pameters | Open Source<br>LLM(DeepInfra) | Commercial<br>API | Open Source<br>MLLM | Commercial API<br>for MLLM |
| --- | --- | --- | --- | --- |
| temperature | 0 | 0 | 0.2 | 1.0 |
| top p | 1.0 | 1.0 | 1.0 | 1.0(GPT4o), 0.8(Qwen-VL-plus) |
| max tokens | 512 | 512 | 512 | 100 |
| presence penalty | 0 | 0 | 0 | 0 |
| frequency penalty | 0 | 0 | 0 | 0 |
| repetition penalty | 1 | 1 | 1 | 1 |

### A.2 Spatial range of 13 Cities

Selected spatial range of 13 cities are presented in Figure 3.

Figure: Figure 3: Maps for Beijing, Shanghai, Cap Town, London, Moscow, Mumbai, Nairobi, San Francisco, Sao Paulo, Sydney, Tokyo.
Refer to caption: x4.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: **(Q1)**: Figure 5: Why is the performance on New York so low, considering it is a very well-known city. Are there any hypotheses for this case?

**(Q2)**: How exactly are satellite/street-view images sampled for the dataset? What is the distribution of these images per city in the benchmark?

**(Q3)**: On examples where the model always provides an answer (i.e. not ones where it refuses to answer), how do the different models compare?

**(Q4)**: Did you try evaluating few-shot performance for these models? How does LLM performance scale with in-context examples?
soundness: 3
strengths: **(S1)**: A comprehensive benchmark with multiple interactive and non-interactive tasks. The authors provide a benchmark that has a good diversity of tasks to evaluate VLM and LLM capabilities. The tasks that form this benchmark are valuable to the research community and for future LLM developers.

**(S2)**: Extensive evaluations with popularly used large language models. The experiments break down the performance for each task for multiple LLMs with sensible metrics. (although I think Llama 405B is missing).

**(S3)**: Publicly released code and API for running the evaluation. It is good that the authors already have the code and API ready for this benchmark, which should aid reproducibility and adoption for CityBench.

Overall, I think the benchmark provided in this paper is valuable to the research community for the diversity of tasks it includes. The paper is well-written and presented, and so I lean towards acceptance.
summary: The paper releases a new systematic benchmark to evaluate LLM and VLM capabilities on geospatial urban data, including interactive and non-interactive tasks that test semantic understanding as well as reasoning.
weaknesses: **(W1)**: Template based questions. The authors specify in section 3.3.1 that they use LLMs to generate instructions for tasks and also use LLMs to filter out low-quality data. More details are needed here, such as which LLMs are used, whether a mixture of LLMs are used, and how much this biases the downstream benchmark performance in favor of certain models. In general, more specific details on the quality control process are required (eg: how much low-quality data was generated, how much was manually curated etc.). 

**(W2)**: Not enough geographic diversity. While the authors do make an effort to collect data for 13 cities across the globe, as they note in Figure 5, the performance on less renowned cities is worse for most LLMs. I think for a more comprehensive benchmark, the authors could consider including a larger diversity of cities for more fine-grained analysis on LLM capabilities.

**(W3)**: Missing comparisons with human baselines. I am curious to know how well humans perform on some of these interactive tasks and it would be good to include a baseline for human-level performance within the benchmark.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: 1、Vague description of data processing: The quality control section mentions the role of human annotators and states that the authors participated in data filtering and rewriting. However, the standards and proportions for filtering and rewriting are not specified. Including these details would make CityBench more transparent and easier to understand.
soundness: 3
strengths: 1. Open Source: The authors have developed CityBench and CitySim, which involve a substantial amount of engineering effort, and have also made the codebase open-source. This is beneficial for future research in this domain.
2. Rich geographic diversity: CityBench covers 13 cities, providing a rich geographic diversity.
summary: This paper introduces CityBench, a systematic benchmark for evaluating the capabilities of large language models (LLMs) in urban research. The authors develop CityData and CitySim to integrate diverse urban data and simulate urban dynamics, constructing CityBench with eight representative tasks. The findings indicate that while advanced LLMs excel in understanding human dynamics and semantic inference in urban images, they struggle with complex tasks requiring specialized knowledge and high-level reasoning, such as geospatial prediction and traffic control. These insights offer valuable perspectives for the future application and development of LLMs.
weaknesses: 1. Figure presentation issues: The left panel of Figure 5 aims to showcase differences in performance across cities, but lacks a legend to identify different cities.
2. Experiment: 1) Incomplete and lacking insight in Error Analysis: The error analysis section only provides insights into LLM errors, with no analysis of VLM errors. Additionally, the LLM error analysis merely highlights general issues common across benchmarks, such as instruction-following limitations and hallucinations, without offering task-specific insights relevant to CityBench’s urban-focused tasks. A more robust analysis could involve systematically sampling each model’s errors on individual tasks, with detailed analysis and conclusions. 2) Incomplete test results: Both Table 2 and Table 3 present CityBench results. While Table 2 omits LLM results due to the visual input requirement, Table 3 involves purely language-based tasks, making it feasible to test VLMs as well. Including VLM results for these tasks would provide a more comprehensive evaluation.
3. Method：1) Transparency of CityBench data volume: Table 1 provides extensive data details for CityData; however, the main text does not include specific data quantities for CityBench. Additionally, the appendix lacks clear information on the exact number of tasks, providing only approximate figures. 2) Unclear task setup: The distinction between testing LLMs and VLMs separately for the Outdoor Navigation and Urban Exploration tasks is not well-explained. According to the descriptions, these tasks are similar, both involving decision-making. However, only LLMs are tested in one task and VLMs in the other, without a clear rationale.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: Please see weakness.
soundness: 3
strengths: 1. It’s exciting to see the promising potential of methods based on interactive simulators in the field of LLMs for urban applications; this work can provide a meaningful boost to community development.

2. The workload is substantial, covering a wide range of tasks with detailed and thorough experiments.
summary: This paper targets the limitations in current explorations of large language models (LLMs) within the urban domain. It creatively introduces an interactive simulator to construct a multi-task, multi-dimensional benchmark and conducts extensive, comprehensive experiments, yielding intriguing conclusions. The authors release their code to foster the development of the community.
weaknesses: 1. The paper provides a somewhat vague description of the pipeline. For example, in Section 3.3.1, what are the criteria for identifying low-quality data? What are the standards for filtering and rewriting? If data quality remains unsatisfactory after filtering and rewriting, is it filtered again or further processed in the same manner?  
  
    In Section 3.1 CityData, the authors state that OSM data is unsuitable for city data construction and introduce a globally applicable rule-based map construction tool. However, there is no specific description of this tool, which leaves readers uncertain about its details and limits understanding of its functionality.

2. A suggestion: the name 'CitySim' is already in use[1], so choosing a different name might help reduce potential ambiguity.

3. In the construction of the Human Activities Data in Section 3.1, data from the literature published in 2016 was used. Can data from a paper published eight years ago accurately reflect the current state of human activities in cities?

4. There is another work CityEval in [2] published in June 2024, which also evaluates the capabilities of LLMs in urban spaces, and some tasks are similar to those in this work. Has the author considered comparing and discussing the differences between the two?

[1] Zheng, Ou, et al. "CitySim: a drone-based vehicle trajectory dataset for safety-oriented research and digital twins." Transportation research record 2678.4 (2024): 606-621.

[2] Feng, Jie, et al. "CityGPT: Empowering Urban Spatial Cognition of Large Language Models." arXiv preprint arXiv:2406.13948 (2024).

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: a. the foursquare data is from 2016. Why not use more current data? Will the old dataset affect the findings? 
b. it is unclear why the five qualities (the paper then interpreted the theory as six qualities) in “the image of the city” are used for GeoQA. The five qualities are proposed for guiding urban design practices. Could you elaborate on the relation between the five qualities and benchmarking GeoQA?
soundness: 2
strengths: a. the utility of LLM for city-related tasks is an interesting point to explore
b. the benchmarking was done in a somewhat broad dataset with cities in different continents and a number of models are tested
c. the categorization of tasks is interesting and useful
summary: The submission presents a framework to evaluate the capabilities of a series of foundation models, including some text-based LLMs and some multi-modal, say, image-text, models, in city-related tasks. It starts from data collection, data processing, and fianally carrying out tasks and benchmarking. It is no doubt the utility of LLM (in the broad sense including multi-modal) in analyzing human activities and city tasks is interesting. This could potentially shape the future research in LLM at large and the development of city-related LLM. However, the paper have not done a good job in providing concrete enough evidence in the underlying causes and lessons learnt. In addition, some experimental designs are flawed.
weaknesses: a. the underlying causes of the good or bad performance of the models in different settings in terms of cities, models, tasks, are not discussed, only with vague and shallow discussions. This is supposed to be the key point to benefit future research and development. It could be ok that we do not push too much on novelty for this submission (more or less should be a major point for iclr though). However, the shallow explanations of the model performance and their comparison are disappointing. For instance, what is the root cause(s) of the unpleasant performance of LLM in regression tasks? 

Indeed, the authors mentioned that LLM some cities do not have as good performance as other cities. The cause here is only “lesser-known”, without presenting any evidence of the underrepresentation. Other aspects, eg socio-economical, morphological, linguistic aspect, are not discussed. 

The benchmarking paper can be benefitted a lot from deeper analysis and discussion. This is supposed to be the key.
b. LLM is in many ways not as good as, or only on par with conventional end-to-end or task specific methods. This somehow is interesting for developments in urban computing. But what is the lesson learnt here is rather sparsely mentioned.
c. the citydata module is only the introduction of datasets used, and it should not be listed as as a contribution brought up by this paper.

---

confidence: 4
contribution: 4
initial_rating: 10
presentation: 4
questions: 1. **Task Diversity**: One of the paper’s key strengths is the wide range of urban tasks included in CityBench. However, do the authors have plans to extend CityBench to cover tasks related to more social or environmental aspects of urban life, such as community planning or disaster response? Expanding the diversity of tasks could further enhance the platform’s applicability across various urban domains.

2. **Geospatial Bias**: The paper highlights the geospatial bias present in VLMs when evaluated across different cities. Could the authors provide further insight into why this bias exists? Specifically, what are the potential underlying factors (e.g., differences in urban morphology, data availability) contributing to this bias, and are there plans to mitigate it in future iterations of CityBench?

3. **Model Performance Variability**: The variability in performance across different LLMs and VLMs was an interesting finding, particularly the fact that larger models did not always outperform smaller ones. Could the authors elaborate on potential causes for this? Do the authors believe this is due to task complexity, model architecture, or data-specific issues?

4. **Error Handling**: The error analysis was quite revealing, especially the frequency of formatting and logic errors in certain models. Are there any specific strategies the authors would suggest to mitigate these types of errors in future deployments of LLMs and VLMs within urban tasks?
soundness: 4
strengths: 1. **High Level of Innovation**:
    - The paper introduces **CityBench**, a highly innovative and comprehensive evaluation platform. CityBench stands out by integrating multi-modal data and interactive simulations to systematically evaluate the performance of LLMs and VLMs in urban tasks, filling a significant gap in the evaluation benchmarks for urban research.
    - The combination of **CityData** and **CitySim** offers researchers a robust tool that simulates fine-grained dynamics across 13 cities. This large-scale, multi-city, multi-modal simulation environment sets CityBench apart from previous work, which has not addressed this level of complexity and scale.

2. **Comprehensive and Detailed Experiment Design**:
    - The experiment design is meticulous, covering both perception tasks (such as image geolocalization and infrastructure inference) and decision-making tasks (such as traffic signal control and mobility prediction). This creates a thorough evaluation framework that includes both straightforward perception tasks and more complex decision-making scenarios.
    - The authors conducted extensive experiments with 30 different LLMs and VLMs, providing a rich dataset and thorough result analysis, validating CityBench as an effective benchmark for urban research.
    - Additionally, the **error analysis and geospatial bias analysis** demonstrate a deep understanding of model behavior across different environments, offering valuable insights for future improvements.

3. **Practicality and Forward-looking Approach**:
    - The paper goes beyond theory by offering an open-source evaluation platform with easy-to-use APIs, significantly increasing its usability and scalability for further urban research applications.
    - It explores the potential of LLMs in urban research, particularly in dynamic and interactive environments, showcasing the future applications of LLMs in urban planning, traffic management, and other real-world urban scenarios.
summary: This paper introduces **CityBench**, an innovative platform designed to evaluate the capabilities of large language models (LLMs) and visual language models (VLMs) in solving diverse urban tasks. CityBench integrates three core components: **CityData**, which collects and processes geospatial, visual, and human activity data; **CitySim**, a simulator that models urban dynamics like individual mobility, visual environments, and microscopic traffic behavior; and **CityBench**, which offers a comprehensive evaluation of eight urban tasks in two categories — perception & understanding and planning & decision-making.

The authors conducted extensive experiments involving 30 LLMs and VLMs across 13 global cities, evaluating tasks such as geospatial prediction, image geolocalization, traffic signal control, and mobility prediction. The results highlight the potential of LLMs and VLMs in handling urban tasks that require commonsense reasoning and semantic understanding, while also exposing challenges in tasks requiring precise professional knowledge and high-level reasoning, such as geospatial prediction and traffic control.

The paper contributes significantly by providing a scalable and systematic evaluation framework, uncovering strengths and limitations of current LLMs/VLMs in urban settings, and paving the way for further development and application of these models in real-world urban scenarios.
weaknesses: 1. **Limited Task Diversity**:
    - While the paper has designed a rich set of urban tasks, there is room for further expansion to include tasks that cover broader social, economic, and environmental aspects of urban research. Future iterations could consider incorporating more tasks related to community planning or disaster response, which would enhance CityBench’s applicability across a wider range of urban research domains.

Meta-review: The paper proposes CityBench, a comprehensive suite to evaluate the capabilities of LLMs for urban tasks. While acknowledging the importance of LLMs and/or VLMs for understanding the urban tasks under a wide variety of adaptations, there is barely a systematic and scalable evaluation benchmark. The paper is more like a comprehensive evaluation / study report of LLMs into the urban tasks. 

Overall this is a good and well-written paper. Most reviewers aknowledge the good merits of the paper:

- Comprehensive, large-scale, high-quality dataset. Promised to be open-sourced to the community.
- CityData and CitySimu, are two contributed modules that are designed to support multiple applications with easy-to-use APIs.

There are also some potential negative comments around the preliminary manuscript:

- Missing experiments for some results in Table 2, 3. Lack of insight. 
- Important technical details to construct the data.

Authors did a fair enough job to address most concerns raised by reviewers. AC carefully read all the review comment, rebuttal and discussions, and thought the scope of the manuscript might not be appropriate for the time being. In a nutshell, this work is more curated to be an LLM evaluation report, rather than technical findings. There might be limitations on the insight / motivation behind all the observations. 

There are several crucial concerns that need to be raised before further consideration.

(a) Whether to contribute to future research is still in doubt. AC agrees with the opinion that is raised by one of the reviewers that the future potential follow-up research might be limitted. Knowing well all the pros and cons of the LLMs, how to apply to urban tasks is still under-explored. Such a report might be a good advide to decision makers, how would it benefit from the real-applications is not convincing. That being said, AC is not judging this kind of research is not useful; but rather would argue there might be limitted impact for the ICLR or similar venue community.

(b) [most crucial point to affect the decision] how each LLM behave is uncertain, due to the black-box nature of LLMs. As such, there might be limited novelty (technical contribution) if there are few insights / rationale behind the behaviors of LLMs. This is actually raised by **all** reviewers in some extent. To name a few:

- Are there any specific strategies the authors would suggest to mitigate these types of errors in future deployments of LLMs and VLMs within urban tasks?

- what is the root cause(s) of the unpleasant performance of LLM in regression tasks?

- Can data from a paper published eight years ago accurately reflect the current state of human activities in cities?

To sum up, the paper is indeed a good investigation with comprehensive amount of work, which indeed is much appreciated. And yet might not be appriopriate for the audience at ICLR.

Decision: Reject