OpenCity: A Scalable Platform to Simulate Urban Activities with Massive LLM Agents

## Abstract

Abstract Agent-based models (ABMs) have long been employed to explore how individual behaviors aggregate into complex societal phenomena in urban space. Unlike black-box predictive models, ABMs excel at explaining the micro-macro linkages that drive such emergent behaviors. The recent rise of Large Language Models (LLMs) has led to the development of LLM agents capable of simulating urban activities with unprecedented realism. However, the extreme high computational cost of LLMs presents significant challenges for scaling up the simulations of LLM agents. To address this problem, we propose OpenCity, a scalable simulation platform optimized for both system and prompt efficiencies. Specifically, we propose a LLM request scheduler to reduce communication overhead by parallelizing requests through IO multiplexing. Besides, we deisgn a “group-and-distill” prompt optimization strategy minimizes redundancy by clustering agents with similar static attributes. Through experiments on six global cities, OpenCity achieves a 600-fold acceleration in simulation time per agent, a 70% reduction in LLM requests, and a 50% reduction in token usage. These improvements enable the simulation of 10,000 agents’ daily activities in 1 hour on commodity hardware. Besides, the substantial speedup of OpenCity allows us to establish a urban simulation benchmark for LLM agents for the first time, comparing simulated urban activities with real-world data in 6 major cities around the globe. We believe our OpenCity platform provides a critical infrastructure to harness the power of LLMs for interdisciplinary studies in urban space, fostering the collective efforts of broader research communities.
Code repo is available at https://anonymous.4open.science/r/Anonymous-OpenCity-42BD .

## 1 Introduction

Agent-based models (ABMs) were first introduced to urban studies in the seminal work of Thomas Schelling about 50 years ago [], which ingeniously explained how segregation can emerge as the aggregation of individual choices. Compared to black-box predictive models, ABMs offer the unique advantage of explaining the underlying mechanisms behind aggregated phenomena [], *i.e.*, revealing the connections between “micro-motives” and “macro-behaviours.” As a result, ABMs play an important role in many research areas [], including computational social sciences, urban planning and public health. The recent advance of Large Language Models (LLMs) have driven the rise of LLM agents [, ], which leverage LLM’s remarkable capabilities of commonsense reasoning and role-playing to simulate human behaviours. Unlike previous rule-based agents, these emerging LLM agents generate far more realistic human behaviours [, ], and can also explain their inner motives via prompting techniques like chain-of-thoughts []. Therefore, LLM agents hold great potential to harness the power of language models in transforming urban studies.

Despite this promising outlook, LLM agents also face severe challenges of scaling up due to the high computation time. In the pioneering work of Park et al. [] only 15 LLM agents were employed to simulate a small village. One main reason is the prohibitive simulation time, which can be broken down into two parts: on one hand, LLMs are inherently slow due to their enormous model sizes; on the other hand, powerful commercial LLMs are only accessible via APIs, which introduces significant time delay due to network transmission, further slowing down simulation. To make matters worse, the prompt design of urban LLM agents often involve dynamic elements, such as the changing memories and perceived environment [, ]. This important feature prevents the straightforward reuse of simulated behaviors from a small sample of the population [], as LLM agents need to maintain independent memories and experiences, which are essential for simulating a vibrant and diverse urban population.

In this paper, we present OpenCity, a scalable platform that introduces both system-level and prompt-level optimizations to enable efficient LLM agent simulation in urban environments. Specifically, we design an LLM request scheduler that leverages the scalable I/O event notification mechanism in operating system (*e.g.*, epoll in Linux []) to minimize network transmission delay. This design is based on our key observation that sending LLM requests and receiving generated output account for only a small portion of total communication time, while the rest are wasted on waiting for LLM responses and the repeatedly establishing TCP connections []. To address this problem, the LLM request scheduler uses the scalable I/O event notification mechanism to parallelize LLM requests by reusing the network I/O portal and TCP connections while waiting for responses.
Besides, LLM request scheduler also analyzes the interdependencies of LLM requests and local computation tasks, *e.g.*, updating agent’s memory and retrieving nearby locations, ensuring local computation tasks are optimally distributed across multiple CPU cores. These system-level optimizations enable large-scale LLM agent simulations to run on commodity hardware. As for the prompt-level optimization, OpenCity introduces a novel “group-and-distill” prompt strategy to minimize the input token required by LLMs. The key idea is to identify the clusters of LLM agents that share semantically similar static elements, *e.g.*, age, gender and income level, and use shared context in batch prompting [] to reduce token redundancy. Specifically, our “group-and-distill” strategy leverages the in-context learning capabilities of LLMs to implement a prototype learning workflow that automatically discovers clusters of LLM agents with semantically similar static elements for simulation. Agents within the same clusters are grouped into a batch prompt, and we design a “prompt distillation” to extract shared prefix for grouped agents. Finally, OpenCity also features an easy-to-use web portal that facilitates code-less simulation configuration and result visualization. This design minimizes the program requirement for running simulation with LLM agents, ensuring our OpenCity platform can benefit researchers from all background.

We evaluate the efficiency and faithfulness of OpenCity in simulating the urban activities of 6 cities around the world using the widely adopted Generative Agent workflow []. Our experiments show OpenCity achieves an average 635x acceleration in simulations with 10,000 LLM agents. Besides, the number of requests and consumed tokens are reduced by 73.7% and 45.5%, respectively. OpenCity also shows strong scalability, with the simulation time per agent reducing from 36.25 to 0.06 seconds as the simulation size increases from 1 to 10,000 agents, demonstrating that larger simulations allow for more efficient LLM request scheduling and prompt distillation. More importantly, OpenCity also maintains high faithfulness of the simulated behaviour. Specifically, the Jensen–Shannon divergence and top-1 hit rates of our method are comparable to the standard prompting technique of batch prompting [], and substantially surpass straightforward reusing strategy []. Besides, the top-1 hit rate can reaches up to 96% when using powerful LLMs like GPT-4o.

The substantial simulation acceleration allows us to benchmark LLM agents’ ability to replicate large-scale urban activities for the first time. We use classic evaluation measures such as the radius of gyration [], origin-destination matrix [], and segregation index [] to assess LLM agents’ simulations. These are the most widely adopted metrics that characterize urban residents’ activities at both individual and group levels, and across physical and social domains. Our experiments show that LLM agents perform comparably to, or better than, traditional rule-based agents like EPR []. Moreover, LLM agents enable counterfactual analyses, such as evaluating experienced segregation in cities without residential segregation []. They also allow researchers to interrogate LLM agents’ motives behind their behaviors, offering valuable insights for urban policy-making.

We believe our OpenCity platform can serve as a critical infrastructure to unleash the power of LLMs in the interdisciplinary studies in urban space. It not only provides high speedup that allows large simulation to run on commodity hardware, but also offers a user-friendly web portal that allows researchers with minimum programming background to access this technology. It will facilitate a broader research community and other stakeholders to use LLM agents to evaluate, analyze and inform their projects.

## 2 Related Works

### 2.1 LLM Agents

With the widespread use of large language models (LLMs) in various applications, the limitations of LLMs, e.g., unstable reasoning abilities, limited memory capacity, and lack of specialized expertise, have been exposed to the public. As one of the potential solution, LLM agents are proposed to overcome these limitations and promote the practical application of LLMs. AutoGPT [] as one of the most popular LLM autonomous agent explore the potential of applying LLM to enable the autonomous planning and task-solving. After that, LLM agents [, ] have made significant progress in two directions: task-oriented agents and simulation agents. Following the first direction, researchers aim to improve LLM agent’s ability to solve complex tasks. For example, lots of programming agents, such as ChatDev [], SWEAgent [], and MetaGPT [], are designed to solve the complex programming tasks. As for the second direction, generative agents [] have demonstrated the potential of large models in simulating human behavior, which has been further validated in subsequent research. S3 [] explores the potential of using LLM agents to simulate the social network. CoPB [] defines a agentic workflow to simulate the mobility behaviors. RecAgent [] simulate the user behavior in the recommendation system. While these works demonstrate the potential of LLM agents, the large scale efficient simulation of generative agents becomes the critical bottleneck of further applications.

### 2.2 LLM Deployment Optimization

To support the efficient inference of LLMs and LLM agents, enormous works and systems [] are designed to optimize the inference efficiency of LLMs and further accelerate their practical applications. For example, Flash-attention [] is an IO-aware exact attention algorithm which uses tiling to reduce the number of memory reads/writes within GPU. AWQ [] is an activation-aware weight quantization to compress and accelerate the LLM inference. vLLM [] proposes pagedAttention mechanism to enable highly efficient KV cache scheduler during the inference and becomes the most population open source LLM inference engine. SGLang [] provides a flexible frontend language to enable the efficient autonomous optimization of LLM inference. Synergy-of-thought [] proposes to exploit the synergy between larger and smaller language models for efficient reasoning. While these systems are designed to process the general LLM inference, specific characteristics of generative agents especially urban generative agents are ignored which can be employed to further accelerate the inference and simulation. In this paper, we explore the potential of this direction and design the OpenCity platform.

## 3 Preliminaries

### 3.1 LLM Agents for Urban Activities

We focus on using LLM agents to reproduce urban dynamics characterized primarily by physical mobility. Consider an urban environment $E$ containing $N$ LLM agents. The state of agent $i$ at simulation time $t$, denoted as $S_{i}(t)=\{s_{i},m_{i}(t)\}$, consists of both static properties $s_{i}$ and dynamic properties $m_{i}(t)$. Static properties, like the agent’s demographics, remain constant throughout the simulation, while dynamic properties, such as memory and perceived environment information, change frequently and are hard to predict.
We can represent the state update of agent $i$ using a function $f$:

$$ $m_{i}(t+1)=f(s_{i},E,m_{i}(t);S_{i}(t+1)=\{s_{i},m_{i}(t+1)\}$ (1) $$

Here, $m_{i}(t+1)$ is the updated memory of agent $i$ at time $t+1$, and the function $f$ models how the agent updates its state by perceiving the urban environment $E$, reflecting on its memory $m_{i}(t)$, and interacting with the LLM.
The individual trajectory of agent $i$, denoted as $T_{i}$, describes the trajectory of the agent over time in the urban environment $E$. If the location of agent $i$ at time $t$ is represented by $L_{i}(t)$, which depends on its state $S_{i}(t)$, then the individual trace can be expressed as $T_{i}=\{L_{i}(0),L_{i}(1),L_{i}(2),...,L_{i}(t_{s})\}$, where $t_{s}$ is the the total simulation time. Along with individual mobility, we also examine the aggregated mobility features $A=\Phi(\sum_{i}{\phi(\sum_{t}S_{i}(t)}))$, such as Original-Destination (OD) matrix and income segregation index, which reflects the urban dynamics involving states of all agents.

To simulate LLM agents in the urban space, we set the initial state for the agents and environment $\{S_{i}(0)|i\in N\}$ and then apply the Equation 1 for each agent at every simulation step. When the number of agents increases, challenges arise mainly because of the LLM request process. LLMs are inherently slow due to their parameter size, and when using commercial LLMs accessed via APIs, response times can be further delayed, especially with poor network conditions. Some have proposed reusing the LLM response for agents can improve the efficiency [], but it requires that the agents have the completely same state or have limit kinds of state that can be easily predicted. What’s more, simply reusing the response would eliminate the independence of agents and reduce the faithfulness of the simulation results. Urban agent $i$ has dynamic memory $m_{i}(t)$ that evolves during the simulation. This memory $m_{i}(t)$ depends not only on past memory $m_{i}(t_{h})$ but also on the current environment. Since decision-making and memory updates rely heavily on the LLM, predicting an agent’s future state or finding an agent with an identical state to reuse an LLM response is difficult. Therefore, to simulate the large-scale and reliable LLM agents for urban dynamics, a simple response reuse strategy is insufficient.

### 3.2 Time Cost Analysis

In light of the prevailing dominance of remote LLM service invocations in the current operational landscape of LLM agents simulation, a decomposition of the time required for a single LLM request can be undertaken, as illustrated in Fig.1(b). The first phase is the initialization and reception time for the LLM request, the second is the TCP/IP connection and destruction time between the simulation system and the LLM service provider, and the third is the data transmission and waiting time. For a single LLM request, the overhead of the first and second phases is relatively low in comparison to the third, and the core time consumption is derived from the data transmission and waiting.

The simulation of large-scale agents necessitates the issuance of a considerable number of LLM requests, which, given the presence of waiting periods, impairs the overall efficiency of the simulator. Furthermore, the system resources are not fully utilized. Consequently, the effective scheduling of LLM requests is essential for enhancing the overall utilization of system resources, which in turn improves the overall efficiency of the simulation. Furthermore, as the time required for LLM inference is directly proportional to the number of tokens contained in an LLM request, it is also important to reduce the number of tokens consumed per agent while compressing the number of requests.

From this vantage point, the present work puts forth an efficacious LLM request scheduler and a prompt distillation method, which can markedly enhance the efficiency of large-scale LLM agents simulation.

## 4 OpenCity Platform

We devise a scalable platform OpenCity to accelerate the simulation of urban LLM agents from both system- and prompt-level. The OpenCity platform aims to substantially reduce the simulation time per LLM agent while maintaining high simulation fidelity. Besides, OpenCity also provides a user-friendly web interface to facilitate the easy access of researchers from diverse background. The key designs are introduced as follows.

### 4.1 LLM Request Scheduler

As shown in Fig.1(a), for a LLM agent, the dependency between its LLM requests—that is, the necessity for the next LLM request to be initiated after the previous one is completed—results in a constant waiting time under the condition of a fixed network environment and request content. In contrast, for a system comprising multiple agents, there is no dependency between their LLM requests. In order to achieve asynchronous processing of multiple LLM requests, we have implemented an IO multiplexing scheme (based on epoll in Linux) which eliminates waiting time in the simulation system. This allows the operating system to manage IO waiting, thereby achieving the desired "zero-awareness" of data transmission in the simulation system. Consequently, the average time for a LLM request is reduced to the time required for the first and second phases (Time saving#1 in Fig.1(c)).

Figure: Figure 1: The functionality of the proposed LLM Request Scheduler.
Refer to caption: extracted/5910015/scheduler.png

Furthermore, the considerable number of LLM calls necessitates the frequent establishment of connections with the service provider, resulting in a considerable overhead in the establishment and destruction of each connection. However, given that the content of LLM requests is inherently linked to the corresponding agent, it is possible to leverage the same connection for multiple agents, thereby reducing the overall performance overhead. To address this issue, a pool of reusable connections is maintained within the system. Upon initiation of an LLM request by an agent, the request content is populated into an available connection, thus avoiding the establishment of a new connection. This approach additionally reduces the mean time consumption of LLM requests (Time saving#2 in Fig.1(c)).

For those agents with CPU tasks during the computation process, it is important to note that the continued occupation of CPU resources by the computation load will inevitably result in a delay in the sending of LLM requests from subsequent agents. To mitigate the adverse effects of this issue on the system’s overall performance, we categorize the CPU task as "local IO", offload it to available cores for computation through a multi-core parallel scheme, and then return the result to the designated agent upon completion of the computation. This approach further ensures the stable operation of asynchronous LLM requests (Time saving#3 in Fig.1(c)).

The proposed LLM request scheduler is designed to reduce the waiting time for a significant number of LLM requests during the simulation runtime. Based on the supporting auxiliary scheme, it has the potential to significantly enhance the efficiency of large-scale LLM agents.

### 4.2 Group-and-Distill Meta-Prompt Optimizer

A further crucial method for enhancing the efficiency of the simulation is to reduce the number of LLM requests issued by agents and the quantity of tokens consumed by said agents. A conventional approach is to reuse the generated result of a single LLM request across multiple agents. However, this approach presents two significant drawbacks: 1. In fine-grained urban LLM agent simulations, each agent possesses its own dynamic properties. Consequently, the reuse scheme compromises the independence of agents, which is antithetical to the objective of conducting urban simulations through large-scale LLM agents. Furthermore, for agents with dynamic properties, it is inherently impossible to share the result of a single LLM request, as shown in Fig.2(a).

Figure: Figure 2: Overview of Group-and-Distill Meta-Prompt Optimizer.
Refer to caption: extracted/5910015/IPL.png

To address this issue, we propose the Group-and-Distill Meta-Prompt Optimizer (depicted in Fig.2), which employs group information in lieu of the static attributes of the agent. This approach aggregates requests from multiple agents at runtime and realizes prompt by sharing group information and context information while preserving the agent’s dynamic properties. The optimizer is comprised of two distinct components. In-context prototype learning (IPL) and distill meta-prompt.

The inputs and outputs of IPL are defined as follow:

$$ $IPL(\{s_{i}\},M,T)\rightarrow\textbf{G},\textbf{D}$ (2) $$

in which, $\{s_{i}\}$ is the collection of agent’s static properties; $M$ controls the number of agents in initial prototype learning; $T$ is the threshold for decision making; G is the collection of agent groups; D is the descriptions for each group of agents.

Input the static properties of a set of agents, IPL first groups the first $M$ agents, providing both group results and the corresponding description information. Subsequently, IPL classifies the remaining agents by transmitting the static properties of the agent to LLM, which analyzes the likelihood of the agent belonging to each group based on the group description and provides the quantization result. By comparing the quantization result with $T$, when the result is greater than $T$, IPL assigns the agent to the specified group. Otherwise, it constructs a new group and describes the characteristics of the group. In comparison to conventional prototype learning methods that operate within a fixed parameter space, IPL exhibits enhanced generalization capabilities and a particular aptitude for leveraging semantic-level knowledge in the prototyping process. The prototype information obtained by IPL is employed to efficiently summarize the static attribute characteristics of the set of agents within the specified group.

The distill meta-prompts obtained through a systematic examination of the original prompts and the CoT approach is employed to generate the prompts (details can be found in Fig.A1). To facilitate the generation procedure, we have proposed a raw prompt design diagram, which divides the prompt into three sections: the function section, the variable section, and the input section. The generation process, which is initiated with a given raw prompt, comprises four steps: summarization, context extraction, information sharing, and rewriting of the raw prompt into the distill meta-prompt. In the operational phase, the requests from the agents in a group are aggregated into a single Distill request, which has the effect of reducing the number of LLM requests and the consumption of tokens.

The proposed prompt optimizer enables further enhancement of simulation efficiency and reduction of simulation cost while maintaining agent dynamic properties.

### 4.3 Web Portal

A web portal has been designed for the utilisation of OpenCity, encompassing the frontend, backend, and simulation system. This enables users to rapidly configure simulation conditions and visualise simulation results, as well as facilitating the storage of simulation data and urban infrastructure information within a database. The fundamental concept underlying the design of this portal is user-friendliness, particularly given the inherently interdisciplinary nature of urban research. We have developed a rapid, code-free configuration approach tailored to the needs of researchers, thereby facilitating the seamless engagement of experts from diverse fields with our simulation platform.

User-friendliness: In order to enhance the usability of the OpenCity platform, the Web Portal has been augmented with the incorporation of the LLM agent blueprint construction function. Users are able to drag and drop each basic function module in order to construct complex logic for LLM agents. In order to meet a variety of needs, the blueprint function is based on the established LLM agent development frameworks, such as Langchain [] and AutoGPT [], and incorporates several fundamental modules oriented towards urban simulation, including environmental and traffic sensing. The blueprint offers an efficient and agile development solution for interdisciplinary researchers, facilitating the rapid iteration of simulation methods and theories.

Basic workflow: The primary process of urban LLM agent simulation on this web portal is comprised of three distinct phases: citizen profile configuration, deployment and simulation, and results presentation. The configuration of the citizen profile is facilitated by the provision of a console hub, which enables users to efficiently and transparently administer the simulation tasks they have created on the platform, along with the agents within those simulations. The user is able to bind the execution logic designed in the blueprint to different agents and to configure their profiles with great rapidity via the web interface. This may entail selecting a city, selecting an existing profile, or filling out a profile manually. Once the configuration process is complete, users can deploy and initiate simulations on the platform with a single click, leveraging the backend system and simulation system. The web portal also offers a monitor page, which enables users to observe the real-time outcomes of ongoing simulations and assess the performance of their agents. Finally, after the simulation has concluded, users can access the portal to view macroscopic statistical results in a visual format, such as Origin-Destination (OD) maps.
An exemplar of the proposed web portal in operation can be found in Figure A2.

## 5 Benchmark

### 5.1 Dataset and Setup

Dataset
We collect urban mobility data in 6 major cities around world: Beijing, New York, San Francisco, London, Paris, and Sydney. The data sources vary. Beijing’s data comes from a related work [], which collected from social network platform. New York and San Francisco source from Safegraph for aggregated population flow data. And the other three cities are from Foursquare which consist of thousands of check-ins data. To make better use of these data, we have done some preprocess method, such as trajectory filter, home extraction and profile sampling. More details can be seen in Appendix A.

Architecture of LLM Agent The main agent used in OpenCity platform to simulate the urban dynamic is the generative agent [].
Generative agents use a framework that involves perception, planning, and reflection. A generative agent first creates a daily plan to ensure the trajectory is reasonable. When the agent arrives at a POI, it makes decisions based on current perceptions and memory. After taking action, the agent records the action and the POI into its memory stream. Once the memory stream reaches a threshold, the agent reflects. The results show that the generative agent to function well in the OpenCity platform.

We also have rule-based agent for comparison, such as the famous Explore and Preferential-Return (EPR) model []. This work make agent choose to explore a new location or return to the visited location. Decisions are related to some parameters to compute the probability. In this paper, we set the parameters as follows: exploration rate $\rho=0.6$, exploration-return trade-off parameter $\gamma=0.21$, waiting time distribution parameters $\tau=17,\beta=0.8$.

### 5.2 Acceleration Performance

This section presents an evaluation of the performance of the OpenCity platform in conjunction with the Generative Agent (Tested on Huawei ECS Cloud Server - Intel(R) Xeon(R) Platinum 8378C CPU @ 2.80GHz with 64 cores and 256 GB RAM). The performance of the platform was evaluated in six major cities with 10,000 agents. The results are presented in Table.1, where the following variables are defined: $Speedup$ denotes the improvements in simulation time, $Rr$ denotes the LLM request number reduction rate, and $Tr$ denotes the token number reduction rate.

The results demonstrate that OpenCity exhibits substantial acceleration in all test cities, with an average runtime of 0.058s per LLM agent and an average speedup of 635.3x in simulation time. Furthermore, the proposed acceleration scheme is capable of markedly reducing the number of LLM requests and token consumption, with an average reduction of 73.7% and 45.5%, respectively.

To assess the scalability of OpenCity, we conducted a series of simulations to evaluate its acceleration performance under varying orders of magnitude of agents. The results of this analysis are presented in Fig.3, in which the baseline represents the simulation time without optimization. The results demonstrate that OpenCity’s acceleration capability is scalable, with a notable enhancement in acceleration effect when the number of agents is increased from 10 to 10,000. This is due to the fact that as the number of agents increases, the number of groups obtained based on IPL also gradually increases. This, in turn, allows the advantages of the LLM request scheduler to be fully realised, thereby ensuring a better utilisation of system resources.

Figure: Table 1: Acceleration experiment results
Refer to caption: extracted/5910015/Scalability.png

Furthermore, faithfulness experiments are conducted to demonstrate that the Group-and-Distill optimizer can effectively preserve the distinctive personality traits of the agents. The testbed for this evaluation is location choice generation, which requires the combination of agent properties to select the next location to visit. A comparison was conducted between the performance of four distinct methods, including raw prompting (without any modification), batch prompting [], archetype prompting [], and the proposed method. One hundred agents were randomly selected and location selection was performed 100 times for each agent with the same context. The effectiveness of the method was evaluated by counting the distribution of selections ($JSD$) as well as the top-1 hit rate ($T1$). The results are shown in Table.2, where Inherent denotes the bias present in LLM itself (raw prompt method).

**Table 2: Faithfulness experiment results**
| Model and Cities | Inherent | Batch prompting | Archetype prompting | Ours |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $JSD$ | $T1$ | $JSD$ | $T1$ | $JSD$ | $T1$ | $JSD$ | $T1$ |  |  |
| 4o-mini | BJ | $0.04\pm 0.02$ | 90% | $0.11\pm 0.05$ | 76% | $0.89\pm 0.04$ | 8% | $0.13\pm 0.02$ | 74% |
| NY | $0.02\pm 0.01$ | 92% | $0.07\pm 0.03$ | 81% | $0.84\pm 0.11$ | 13% | $0.06\pm 0.04$ | 86% |  |
| SF | $0.03\pm 0.02$ | 88% | $0.09\pm 0.04$ | 77% | $0.91\pm 0.03$ | 11% | $0.10\pm 0.03$ | 85% |  |
| Lo | $0.06\pm 0.04$ | 89% | $0.12\pm 0.07$ | 79% | $0.86\pm 0.06$ | 9% | $0.12\pm 0.04$ | 78% |  |
| Pa | $0.05\pm 0.02$ | 86% | $0.17\pm 0.11$ | 69% | $0.94\pm 0.03$ | 4% | $0.14\pm 0.04$ | 71% |  |
| Sy | $0.04\pm 0.03$ | 85% | $0.08\pm 0.03$ | 75% | $0.88\pm 0.05$ | 5% | $0.07\pm 0.04$ | 75% |  |
| GPT-4o | NY | $0.003\pm 0.002$ | 98% | $0.012\pm 0.007$ | 94% | $0.89\pm 0.09$ | 10% | $0.009\pm 0.004$ | 97% |
| Pa | $0.004\pm 0.002$ | 99% | $0.021\pm 0.009$ | 93% | $0.91\pm 0.04$ | 7% | $0.010\pm 0.006$ | 96% |  |

As evidenced by the results, our method demonstrates the capacity to maintain a comparable level of consistency to that observed in the batch prompting method, while exhibiting a reduction in volatility and token consumption. However, the archetype prompting method performs poorly in this evaluation, which further demonstrates the inability of the reuse-based method to accommodate the dynamic properties of agents. Furthermore, given the considerable discrepancies observed in the raw prompting method when evaluated using the GPT-4o-mini model, an additional assessment was conducted on two cities, New York and Paris, utilising the GPT-4o model. The findings indicate that our method is capable of approximating the execution of the raw prompting method to a significant degree. Additionally, the results indicate that there are notable discrepancies between different models in terms of environmental comprehension and the capacity to process lengthy textual content. The consistency of LLM outcomes merits further examination.

In general, OpenCity is capable of markedly enhancing the efficiency of large-scale urban LLM agent simulations while concurrently preserving the distinctive characteristics of the agents themselves. This enables the cost of simulating populations exceeding 10,000 to be maintained at the hourly level.

### 5.3 Reproducing Urban Dynamics

The significant increase in simulation efficiency enables us to benchmark LLM agent’s ability to reproduce large-scale urban dynamics for the first time. We use comprehensive metrics in three-levels to evaluate the simulation performance, from individual- to group level, and also from physical domain to social domain. At the individual level, we calculate the radius of gyration [] for each user. At the group level, we use the original-destination matrix []. As for the social domain, we focus on the income segregation index []. To evaluation the simulation performance, we compute the MSE for these three metrics, which are denoted as $R_{MSE}$, $OD_{MSE}$ and $S_{MSE}$. More details can be referred to Appendix B.

In this section, we analyze the performance of the Generative Agent and EPR Agent in reproducing urban dynamics. We test both agents in 6 major cities using 1,000 agents. The results are shown in Table 3. The results indicate that both the Generative Agent and EPR Agent successfully reproduce urban dynamics with low MSE values. Additionally, the LLM Agent performs as well as or better than the classical rule-based EPR Agent, highlighting the advantage of LLM’s semantic understanding ability in urban simulations.

**Table 3: Urban dynamics reproduction results**
| Cities | GenerativeAgent | EPR |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| $R_{MSE}$ | $OD_{MSE}$ | $S_{MSE}$ | $R_{MSE}$ | $OD_{MSE}$ | $S_{MSE}$ |  |
| Beijing | 19.5 | 3.88e-4 | 0.0312 | 29.8 | 4.26e-4 | 0.0630 |
| New York | - | 5.95e-4 | 0.3521 | - | 3.70e-4 | 0.2319 |
| San Francisco | - | 23.6e-4 | 0.1535 | - | 14.0e-4 | 0.0352 |
| Paris | 2.48 | 7.58e-4 | 0.1255 | 4.04 | 6.25e-4 | 0.1240 |
| London | 6.24 | 5.22e-4 | 0.1258 | 25.7 | 7.41e-4 | 0.1501 |
| Sydney | 15.1 | 4.71e-4 | 0.1118 | 54.2 | 7.63e-4 | 0.1265 |

## 6 Case Study: Experienced Urban Segregation

With the ability to simulate large-scale urban LLM agents, we can conduct counterfactual experiments to explore outcomes under different policies and design optimal strategies for the future. Conventional rule-based models do not support this capability, as they are designed to simulate real-world scenarios.
Experienced urban segregation is a widely discussed issue with significant impacts on social dynamics and the economy. It arises from both demographic differences in residential neighborhoods and the mobility patterns of urban residents [].

This section provides a case study: a counterfactual simulation is conducted in New York and San Francisco, to observe how the simulation results change in different configurations, and try to summarize the results with the LLM agents themselves.

Specifically, we construct the counterfactual scenario by evenly distributing LLM agents with different income levels across the city, that means we almost eliminate the residential segregation. The results of the income segregation statistics with CBGs as the statistical granularity are shown in Fig.4, where ‘Original’ samples the segregation results from the real census data, and ‘Even’ is the result after uniform distribution of agents with different incomes.

Figure: Figure 4: The distribution of income segregation index for counterfactual experiment.
Refer to caption: extracted/5910015/counterfactual.jpg

From the results, it can be seen that the segregation of the two cities changed significantly after the different income groups were evenly distributed in the cities. In New York City, the mean segregation index decreases from 0.845 to 0.172, and in San Francisco, the mean segregation index decreases from 0.665 to 0.232.
As a result, we believe that differences between regions are the main cause of segregation as opposed to segregation by choice of action.
To extend, we can know that with policies that promote more even income distribution among neighborhoods, urban segregation and social inequality can be improved.

Figure: Figure 5: A detail case of interpreting simulation results through communication.
Refer to caption: extracted/5910015/Case.png

Furthermore, we use natural language to communicate with those involved agents to gain deeper insights about urban segregation. One detailed case is shown in Fig.5. When we ask an agent about its daily journey, it can accurately provide the time and locations it visited. This is because the agent caches runtime information and uses the LLM’s ability to understand semantic details. When asked about the people it met, the agent lists everyone it encountered at different locations and provides their information. This is due to vectorized storage of the agent’s simulation results and the LLM’s ability to retrieve that information. Collecting and observing fine-grained statistical information through conversations with agents and even through LLM improves both the interpretability of the simulation and our understanding of the simulation goals.

## 7 Conclusion

In this work, we introduced OpenCity, a scalable platform designed to address the computational and communication challenges inherent to the deployment of large-scale LLM-based urban agents in city simulations. By incorporating an LLM request scheduler and a novel "group-and-distill" prompt optimization strategy, we achieved a notable 600-fold increase in the efficiency of agent simulations, with a substantial reduction in both LLM requests and token usage. The OpenCity platform was evaluated through experiments conducted on six global cities. The results demonstrated the platform’s capability to simulate the daily activities of 10,000 agents at an hourly level, while also establishing a benchmark for generative agent performance in urban contexts. The platform’s ability to compare simulated behaviors with real-world data highlights its potential for real-world urban-scale applications, offering a robust tool for urban planners and researchers to explore and understand complex societal phenomena.

## Appendix A Urban Mobility Dataset

As shown in Table A1, we collect urban mobility data of 6 major cities around the world. The data sources vary. In Beijing, the data is from a related work [], which gathered through a social network platform and tracking users’ mobility trajectories. Additionally, users’ profiles, such as income level, gender, occupation, education level and age, are collected through digital surveys. In New York and San Francisco, the data comes from Safegraph, which provides aggregated population flow among Points of Interest(POIs) and Census Block Groups(CBGs). The other three cities—London, Paris and Sydney—use data are from Foursquare. Foursquare data consist of thousands of check-ins data of users and the corresponding venue position.

To make better use of the datasets, we apply several preprocessing methods. We firstly arrange the trajectory points in time sequence, and divide the trajectory into units of one day. Then we filter out trajectories with fewer than 4 points in a day, as they do not fully capture users’ mobility patterns. For home extraction, we identify the most frequently visited location of the useres the home. Since only the Tencent dataset includes user profiles, we make profile sampling for users of each city based on local census data for the other two datasets. In the end, our dataset is optimized for easy use in urban mobility simulations.

**Table A1: Basic information about the dataset**
| Source | City | Users | Trajectory Points | Duration |
| --- | --- | --- | --- | --- |
| [] | Beijing | 100000 | 297363263 | Oct. 2019 - Dec. 2019 |
| Safegraph | New york | Aggregated | 760493 | May 2023 - July 2023 |
| San Francisco | Aggregated | 316732 |  |  |
| Foursquare | London | 9409 | 173268 | Apr. 2012 - Sept. 2013 |
| Paris | 5809 | 85679 |  |  |
| Sydney | 1720 | 54170 |  |  |

## Appendix B Urban dynamic metrics

We use comprehensive metrics in three-levels to evaluate the simulation performance, from individual-level to group level, and also from physical domain to social domain. These metrics allows us to gain a full understanding of mobility patterns and their implications, and can also help us evaluate the performance of the simulation by analysing the generated trajectory.

At the individual level, we calculate radius of gyration $r_{g}$ [] for each user, which is a measure of the spatial extent of their movements. The radius of gyration is defined as follows:

$$ $r_{g}^{\alpha}=\sqrt{\frac{1}{N^{\alpha}}\sum_{i=1}^{N^{\alpha}}{(\vec{r}_{i}^ {\alpha}-\vec{r}_{cm}^{\alpha})^{2}}}$ (3) $$

where $\vec{r}_{i}^{\alpha}$ represents the $i=1,2,...,N$ positions recorded by user $\alpha$, and $r_{cm}^{\alpha}=1/{N^{\alpha}}\sum_{i=1}^{N^{\alpha}}{(\vec{r}_{i}^{\alpha})}$ is the center of mass of the trajectory. The radius of gyration provides an indication of the size of a user’s activity range. To assess the accuracy of our simulation data against real-world data for a specific user, we calculate $R_{MSE}$, the Mean Squared Error(MSE) of the radius of gyration.

To analyze movement patterns and other aggregated features, we define block areas as spatial units within the city. For cities with Safegraph data, we use existing Census Block Group (CBG) areas. For other cities, we divide the map area into evenly spaced grids, with each grid cell representing a block area.

At the group level, we count the inflow and outflow of agents between block areas, calculate the Origin-Destination (OD) matrix [], and normalize it. To compare real data with simulation data, we calculate the MSE of the normalized OD matrix, denoted as $OD_{MSE}$. A smaller $OD_{MSE}$ value indicates greater similarity between the OD matrices, meaning the movement characteristics of the simulated data closely match the real data.

At the social domain, we calculate the income segregation index [] for each block area. The income segregation of a place $\alpha$ is defined as $S_{\alpha}=\frac{5}{8}\sum_{q}{|\tau_{q\alpha}-\frac{1}{5}|}$, where $\tau_{q\alpha}$ is is the proportion of visitors in each income quintile for place $\alpha$. The $S_{\alpha}$ ranges from 0 to 1. A high $S_{\alpha}$ indicates that the place $\alpha$ is predominantly visited by a single income group, suggesting a high level of income segregation. We denote $S_{MSE}$ as the MSE between the real data and simulation data.

## Appendix C Image supplements

Figure: Figure A1: Distill meta-prompt generation through CoT inference.
Refer to caption: extracted/5910015/CoT_SPG.png

Figure: (a) Agent Construction
Refer to caption: extracted/5910015/Blueprint_mini.png

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: While the approach and system are interesting, I don't see this paper as a good fit for ICLR, in its present form, and suggest it be rejected. 
Some of my questions:

- What LLM model is used in the simulation, and how was it trained?
- How do the prompts look before and after applying the group-and-distill approach?
- What is the output of the LLM, and how does it influence agent behaviour?
- To what extent is the group-and-distill technique generalisable beyond urban simulations?
- How does the platform maintain long-term consistency in agent behaviour, given the variability in LLM responses?
- What mechanisms are in place to manage or correct for inconsistencies in agent behaviour across prompts?
- Is there an evaluation of the platform’s accuracy in simulating real-world behaviours compared to traditional ABMs?
soundness: 1
strengths: The use of a LLM for the purpose of larger scale population modelling appears to be novel, and the suggested group-and-distill approach enables this idea, with relatively low hardware resources. 
Overall, considerable effort appears to have gone into development of the system. The system could be an interesting resource for research in complex systems.
summary: The paper describes an approach	where LLM agents are used to simulate individual behaviour in large (city-scale) simulations of people.	The proposed platform uses LLM agents that can adapt their behaviour depending on context and memory. This is different to the classic agent based approach for this type of simulation where behaviours are static over time. 
The development of the platform is one of the main contributions of the work, and the development of a user-friendly web interface is another contribution highlighted by the authors. From a machine learning perspective, the proposed “group-and-distill” approach to reduce LLM usage is the main contribution of the work, essentially a clustering approach before prompting the LLM for each cluster (as opposed to prompting an LLM for each individual).
weaknesses: The paper has quite a broad focus, like an overall project report. For a venue like iclr, it would have been better to focus on the specific contributions in machine learning, and to provide more technical details rather than an overall description of architecture and usability aspects as the main contributions. In its current form, ICLR does not appear to be the right venue for the work as it is presented.

The work lacks depths in aspects that I would see essential for any ML paper: for example the group-and-distill concept is introduced, but the paper is very sparse in detail of the specific algorithms. Similarly it would have been interesting to see what are the initial prompts and the optimised prompts, in contrast. 
Any details comparing to the original approach without group-and-distill / ablation would have been an improvement too. 

Moreover it didn’t become clear to me what LLM has been used or how was it trained, and how do LLM outputs influence agents’ behaviours.

Finally, the paper mentioned at the beginning the explainability of ABM as an advantage over black box neural network approaches. with the lack of detail on how the actions are influenced by the LLM or how the LLM are trained or fine tuned, the proposed model has the same disadvantage as any other neural network model.  

Minor presentation issues:

"Agent-based models (ABMs) were first introduced to urban studies in the seminal work of Thomas Schelling about 50 years ago Schelling (2006)"
- if the work referenced was from approx 50 years ago, Schelling 2006 is the wrong reference. I believe the correct year would be 1978.
- there are two	kinds of citations, narrative (like the one in the sentence), and parenthetical (Schelling, 2006). It doesn't make sense to use	narrative style	when it doesn't fit into the sentence structure. In LaTeX with natbib, this is the difference between \citet and \citep. 
The referencing is an issue throughout the paper.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 2
questions: 1.	Why did the authors conduct additional assessment on merely two cities: New York and Paris using the GPT-4o model in Table 2? Rather than conducting experiments in all six cities like 4o-mini?
2.	As for the experimental setup, do the following parameters: exploration rate ρ= 0.6, exploration-return trade-off parameter γ = 0.21, waiting time distribution parameters τ = 17 affect the results？
3.	In line 389，what does baseline mean? As the citation is missing, the reviewer guess whether it means the method in Park et al. (2023)？If not, comparative experiments on the Park et al. (2023) method should be added.
4.	 Why the result of baseline method is 50s/agent when the number of agents is very small in Figure 3, such as merely a single agent?
soundness: 3
strengths: Strengths
1.	This paper introduces a scalable platform for urban simulations using LLM agents, which addresses a growing need for realistic human behavior modeling in urban environments.
2.	This paper shows a high quality of presentation. The paper is technically sound and the research question is clear. The optimizations, particularly the LLM request scheduler and prompt optimization strategies, demonstrate clear performance benefits. The experimental results showing a 600x speedup and significant resource savings are compelling.
3.	The paper is generally clear and well-structured. It provides a clear problem statement, introduces the proposed framework, and highlights key findings.
4.	The contribution of the paper is relevant for LLM agent. The results of this paper is interesting and significant in LLM agent simulation. The proposed OpenCity framework is relevant for urban planning and policy-making. The development of a web portal that allows researchers to configure and visualize simulations without requiring programming skills is a valuable addition, making the platform accessible to a broader audience.
summary: The recent rise of Large Language Models (LLMs) has led to the development of LLM agents capable of simulating urban activities with unprecedented realism. Nevertheless, the extreme high computational cost of LLMs presents significant challenges for scaling up the simulations of LLM agents. With this motivation, this paper introduces OpenCity, a scalable simulation platform designed to efficiently simulate urban activities using a large number of LLM agents. The platform incorporates innovative techniques, including LLM request scheduler and a group-and-distill prompt optimization strategy, to reduce the computational overhead of simulating LLM agents significantly. OpenCity achieves a 600-fold speedup and reduces both LLM requests and token consumption. Extensive experiments on six global cities verify the platform's scalability and its capability to replicate real-world urban dynamics.
weaknesses: 1.	The introduction part fails to convey to the reviewers what is the motivation and novelty in this paper. In fact, the authors should add more previous work on LLM agents based simulation platform. The problem this paper addresses and the reason why this paper uses system-level LLM request scheduler and prompt-level “group-and-distill” strategy to solve the problem of scalability should be further explained. Besides, the contribution the authors listed in the introduction section is inaccurate，the authors should focus on the system-level LLM request scheduler and prompt-level “group-and-distill” strategy. Thus, I would recommend a revision for the introduction section in this paper.
2.	This paper utilizes Group-and-Distill Meta-Prompt Optimizer to classify similar agents to reduce computational complexity, which indeed improve efficiency. However, this may overlook differences between individuals, so the reason why this method can preserve the distinctive characteristics of the agents, as show in the experimental part, should be further explained in the method section。
3.	Figure 2 illustrates the principle of Group-and-Distill Meta-Prompt Optimizer. However, it seems difficult to follow. It is more intuitive to add an example to explain how IPL works. 
4.	There lacks explanation for the reason why the proposed method IPL is superior to conventional prototype learning. Moreover, the principles for setting the value of M and T in IPL should be further illustrated.
5.	Experimental part: the authors should add an explanation of the indicators including JSD, T1 and bold the important data in Table 2 . Similarly, Table 3 also requires revision. The metrics of RMSE of New York和San Franciscoin are not displayed in Table 3, which seems a little bit confusing, the authors need to provide explanations. 
6.	The authors should pay attention to the standardization of citations throughout the paper, especially in introduction and related works section. For example, “conventional prototype learning methods...”(line305), “the baseline represents the simulation time without optimization” (line 389), “we analyze the performance of the Generative Agent and EPR Agent ”(line 450).
7.	The authors should carefully proofread the manuscript for typos and formatting issues. There exists some typos: in the abstract: “we deisgn a “group-anddistill” prompt optimization”, “where τqα is is the proportion”(line 687) , etc.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: * The paper spends comparatively less effort on explaining what kind of benefits do we expect from an LLM-based ABM. For instance, we can try to model thought processes of the humans, or their communication. Does the choice of this modeling impact the proposed techniques?
soundness: 2
strengths: * The overall goals of the paper, of capitalizing on the abilities of LLMs to achieve a better ABM model of cities, as well as addressing the scaling problems, are laudable.
summary: The paper is considering the problem of agent-based modeling of environments such as cities. Such models had been previously used with the agents performing relatively simple behaviors. While LLMs open new opportunities for controlling the behaviors of the individual agents, their computational cost presents a significant scaling problem. 

The paper describes an architecture that enables the parallelization of the agents, to allow the modeling the daily activities of a city with 10,000 agents. The architecture appears to be based on an efficient polling model of the LLM, as well as the development of a prompting model called "group-and-distill". The application of these models show a more than 600-fold increase.
weaknesses: * Achieving a more than 600 fold speed increase in terms of an improved process scheduler and I/O scheduler can be probably seen as "debugging", rather than research result, and very likely has nothing to do with the LLM. 
* It seems that the very considerable computational effort of an LLM can only achieve an approximate parity with the much cheaper rule based efforts. This is understandable, as description of the behavior of the agents described in the paper follows the same position based rules that the ABM models historically use. As there is no consideration of language or other type of reasoning, the paper does not make it clear what type of benefits one would expect from LLMs.
* The only part of the paper that has a connection to the topic of this conference is the way in which the "group-and-distill" model is proposed to achieve the simulation of multiple agents with one prompt. However, there is very little about this technique in the paper proper, so it is difficult to form a judgement.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1.	It seems that the formula in Equation 1 misses a parenthesis?  And it can also be split into two separate equations.
2.	In Section 3.2, the content of Figure 1(b) is introduced, however, the Figure 1(a) is discussed in Section 4.1. Is this logical? It is suggested to swap the subtitles of the figures for better coherence.
3.	This paper categorized the CPU task as ”local IO”, offload it to available cores for computation through a multi-core parallel scheme, and then return the result to the designated agent upon completion of the computation.
4.	Is there an increase in CPU task scheduling time? How does it compare to the time saved in saving#3?
5.	The paper states that "A conventional approach is to reuse the generated result of a single LLM request across multiple agents, and this approach presents two significant drawbacks." However, it only describes one drawback.
6.	The paper introduces Figure 2(a) first, which prompts readers to check the content of Figure 2(a). However, they will encounter a series of unexplained equations, which could cause confusion. The paper cites Figure 2 only in the subsequent description, and the explanation of Figure 2(a) is found only in the explanation of the methodology of Figure 2 in the main text. Is it possible to optimize this part of the description?
7.	From the subsequent description in the main text, it is understood that Figure 2(a) is introducing the IPL method, but in the figure is labeled as the LPL method.
8.	How is the threshold T in the IPL method of this paper determined?
9.	Are the baseline in Figure 3 increased simulation response time due to the load as the number of agents increases?
10.	In Table 3, there is no explanation for the missing RMSE results for New York and San Francisco. According to Appendix A, the data for these two cities comes from Safegraph, and the number of users is aggregated. So how is the GROUP-AND-DISTILL performed in these two cities?
soundness: 2
strengths: The paper give a detailed introduction of the novel methods and the real outcome.

1. Originality: The paper presents a novel approach by integrating Large Language Models with agent-based modeling to simulate urban activities. There has been limited research on combining LLMs with agent-based models, and even less so in the context of large-scale urban activity simulations. By using IO multiplexing and the "group-and-distill" prompt optimization strategy to reduce the computational cost of simulations, this paper has made the application of LLMs in large-scale urban activity simulations possible.
2. Quality: The research is well organized with a clear methodology and experiments conducted in real cities data. The results show notable improvements in both simulation efficiency and accuracy, confirming the effectiveness of the proposed platform.
3. Clarity: The paper is written in a clear and concise manner; it is easy to understand through the explanation of figures
4. Significance: This paper establishes a benchmark for LLM agent-based urban activity simulation research. This paper also provides a scalable framework for simulating urban dynamics.
summary: ABM and LLM is leveraged to develop one platform for open city modeling and planning. It is a nice simulation platform and the paper provides application scenarios. Concretely, 

1. This paper combines agent-based models with large language models to develop the OpenCity platform for simulating urban activities. It reduces simulation computational costs through IO multiplexing and the "group-and-distill" prompt optimization strategy.
2. Through experiments conducted in six major cities worldwide, OpenCity demonstrates a 635 plus increase in average simulation speed per agent, along with a 70% decrease in LLM requests and a 50% reduction in token usage. The time savings are mainly concentrated in the LLM response wait time and the CPU multiplexing process.
3. The OpenCity platform proposed in this paper achieves the first benchmark testing for LLM agent-based urban activity simulation research.
weaknesses: There is a lack of theoretical contribution, overall, rather it is an application tool development with leveraging well established tools. It may not fit ICLR the best though not out of scope at all. Further,

1. Some parts of the main body text are not rigorous enough. For example, Equation 1 is missing a parenthesis, and the IPL method is mistakenly labeled as the LPL method in Figure 2.
2. This research has high requirements for data quality. Additionally, despite significantly improving computational efficiency and reducing costs, the platform may still require substantial computational resources.
3. When simulating cities in different countries, the dynamic properties to be considered should not be entirely the same, and some of the assumed static properties may also change during the simulation process.

Meta-review: The paper proposes OpenCity, a platform combining agent-based modeling (ABM) with large language models (LLMs) to simulate urban dynamics at scale. By leveraging techniques like the “group-and-distill” prompt optimization, the platform achieves significant computational efficiencies, including a 600-fold speedup and reduced resource consumption in large-scale simulations. The paper claims to establish a new benchmark for LLM-powered urban simulation and provides a user-friendly web interface for broader accessibility.

While the proposed OpenCity model shows promising results, the reviewers have identified several weaknesses that need to be addressed:
1. The paper’s broad focus, which includes platform architecture and usability aspects, lacks sufficient technical depth on the machine learning contributions, especially regarding the “group-and-distill” approach.
2. The absence of detailed algorithmic descriptions and comparisons, such as ablation studies or specifics on optimized prompts, weakens the paper’s technical rigor.
3. There is a lack of clarity on which LLM model is used, how it is trained, and how its outputs influence agent behavior, leaving the model’s inner workings inadequately explained.

Based on these weaknesses, we recommend rejecting this paper. We hope this feedback helps the authors improve their paper.

Decision: Reject