Large Language Models as Urban Residents: An LLM Agent Framework for Personal Mobility Generation

Authors: jiawei  wang
renhe  jiang
chuang  yang
zengqing  wu
makoto  onizuka
ryosuke  shibasaki
noboru  koshizuka
chuan  xiao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 20:44:54

Abstract: This paper introduces a novel approach using Large Language Models (LLMs) integrated into an agent framework for flexible and effective personal mobility generation. LLMs overcome the limitations of previous models by effectively processing semantic data and offering versatility in modeling various tasks. Our approach addresses three research questions: aligning LLMs with real-world urban mobility data, developing reliable activity generation strategies, and exploring LLM applications in urban mobility. The key technical contribution is a novel LLM agent framework that accounts for individual activity patterns and motivations, including a self-consistency approach to align LLMs with real-world activity data and a retrieval-augmented strategy for interpretable activity generation. We evaluate our LLM agent framework and compare it with state-of-the-art personal mobility generation approaches, demonstrating the effectiveness of our approach and its potential applications in urban mobility. Overall, this study marks the pioneering work of designing an LLM agent framework for activity generation based on real-world human activity data, offering a promising tool for urban mobility analysis.

Main paper: 
 introduction: 
 The prevalence of large language models (LLMs) has facilitated a variety of applications extending beyond the domain of NLP. Notably, LLMs have gained widespread usage in furthering our understanding of humans and society in a multitude of disciplines, such as economy [1] and political science [2] , and have been employed as agents in various social science studies [36, 35, 10] . In this paper, we target the utilization of LLM agents for the study of personal mobility data. Modeling personal mobility opens up numerous opportunities for building a sustainable community, including proactive traffic management and the design of comprehensive urban development strategies [4, 3, 45] . In particular, generating reliable activity trajectories has become a promising and effective way to exploit individual activity data [13, 6] . On one hand, learning to generate activity trajectory leads to a thorough understanding of activity patterns, enabling the flexible simulation of urban mobility. On the other hand, while individual activity trajectory data is abundant thanks to advances in telecommunications, its practical use is often limited due to privacy concerns. In this sense, generated data can provide a viable alternative that offers a balance between utility and privacy. While advanced data-driven learning-based methods offer various solutions to generate synthetic individual trajectories [13, 39, 9, 6, 16, 38] , the generated data only imitates real-world data from the data distribution perspective rather than semantics, rendering them less effective in simulating or interpreting activities in novel or unforeseen scenarios with a significantly different distribution (e.g., a pandemic). Thus, in this study, to explore a more intelligent and effective activity generation, we propose to establish a trajectory generation framework by exploiting the emerging intelligence of LLM agents, as illustrated in Figure 1 . LLMs present two significant advantages over previous models when applied to activity trajectory generation:• Semantic Interpretability. Unlike previous models, which have predominantly depended on structured data (e.g., GPS coordinates-based trajectory data) for both calibration and simulation [17, 27, 47] , LLMs exhibit proficiency in interpreting semantic data (e.g., activity trajectory data). This advantage significantly broadens the scope for incorporating a diverse array of data sources into generation processes, thereby enhancing the models' ability to understand and interact with complex, real-world scenarios in a more nuanced and effective manner. • Model Versatility. Although other data-driven methods manage to learn such dynamic activity patterns for generation, their capacity is limited for generation under unseen scenarios. On the contrary, LLMs have shown remarkable versatility in dealing with unseen tasks, especially the ability to reason and decide based on available information [26] . This competence enables LLMs to offer a diverse and rational array of choices, making it a promising and flexible approach for modeling personal mobility patterns.Despite these benefits, ensuring that LLMs align effectively with real-world situations continues to be a significant challenge [36] . This alignment is particularly crucial in the context of urban mobility, where the precision and dependability of LLM outputs are essential for the efficacy of any urban management derived from them. In this study, our aim is to address this challenge by investigating the following research questions: RQ 1: How can LLMs be effectively aligned with semantically rich data about daily individual activities? RQ 2: What are the effective strategies for achieving reliable and meaningful activity generation using LLM agents? RQ 3: What are the potential applications of LLM agents in enhancing urban mobility analysis?To this end, our study employs LLM agents to infer activity patterns and motivation for personal activity generation tasks. While previous researches advocate habitual activity patterns and motivations as two critical elements for activity generation [17, 43] , our proposed framework introduces a more interpretable and effective solution. By leveraging the capabilities of LLMs to process semantically rich datasets (e.g., personal check-in data), we enable a nuanced and interpretable simulation of personal mobility. Our methodology revolves around two phases: (1) activity pattern identification and (2) motivation-driven activity generation. In Phase 1, we leverage the semantic awareness of LLM agents to extract and identify self-consistent, personalized habitual activity patterns from historical data. In Phase 2, we develop two interpretable retrieval-augmented strategies that utilize the patterns identified in Phase 1. These strategies guide LLM agents to infer underlying daily motivations, such as evolving interests or situational needs. Finally, we instruct LLM agents to act as urban residents according to the obtained patterns and motivations. In this way, we generate their daily activities in a specific reasoning logic.We evaluate the proposed framework using GPT-3.5 APIs over a personal activity trajectory dataset of Tokyo. The results demonstrate the capability of our framework to align LLM agents with semantically rich data for generating individual daily activities. The comparison with baselines, such as attention-based methods [8, 22] , adversarial learning methods [6, 42] , and a diffusion model [46] , underscores the advanced generative performance of our framework. The observation also suggests that our framework excels in reproducing temporal and spatio-temporal aspects of personal mobility generation and interpretable activity routines. Moreover, the application of the framework in simulating urban mobility under specific contexts, such as a pandemic scenario, reveals its potential to adapt to external factors and generate realistic activity patterns.To the best of our knowledge, this study is one of the pioneering works in developing an LLM agent framework for generating activity trajectory based on real-world data. We summarize our contributions as follows: (1) We introduce a novel LLM agent framework for personal mobility generation featuring semantic richness. (2) Our framework introduces a self-consistency evaluation to ensure that the output of LLM agents aligns closely with real-world data on daily activities. (3) To generate daily activity trajectories, our framework integrates activity patterns with summarized motivations, with two interpretable retrieval-augmented strategies aimed at producing reliable activity trajectories. (4) By using real-world personal activity data, we validate the effectiveness of our framework and explore its utility in urban mobility analysis.2 Related Work 
personal mobility generation: 
 Activity trajectory generation offers a valuable perspective for understanding personal mobility. Based on vast call detailed records, Jiang et al. built a mechanistic modeling framework to generate individual activities in high spatial-temporal resolutions. Pappalardo and Simini employed Markov modeling to estimate the probability of individuals visiting specific locations. Besides, deep learning has become a robust tool for modeling the complex dynamics of traffic [15, 13, 44, 9, 21] . The primary challenge involves overcoming data-related obstacles such as randomness, sparsity, and irregular patterns [8, 43, 42, 20 Exploring how to treat LLMs as autonomous agents in specific scenarios leads to diverse and promising applications in social science [32, 36, 35, 10] . For instance, Park et al. established an LLM agent framework to simulate human behavior in an interactive scenario, demonstrating the potential of LLMs to model complex social interactions and decision-making processes. Moreover, the application of LLM agents in economic research has been explored, providing new insights into financial markets and economies [11, 19] . Extending beyond the realm of social sciences, Mao et al. adeptly utilized LLMs to generate driving trajectories in motion planning tasks. In the field of natural sciences, Williams et al. integrated LLMs with epidemic models to simulate the spread of diseases. These varied applications highlight the versatility and potential of LLMs to understand and model various real-world dynamics. 
methodology: 
 We consider the generation of individual daily activity trajectories, each representing an individual's activities for the whole day. In addition, we focus on the urban context, where the activity trajectory of each individual is represented as a time-ordered sequence of location choices (e.g., POIs) [21] . This sequence is represented by {(l 0 , t 0 ), (l 1 , t 1 ), . . . , (l n , t n )}, where each (l i , t i ) denotes the individual's location l i at time t i .By modeling individuals within an urban environment as LLM agents, we present LLMob, an LLM Agent Framework for Personal Mobility Generation, as illustrated in Figure 2 as crucial information for inferring daily activities [31, 7, 30] . On the other hand, motivations relate to dynamic and situational elements that sway an individual's choices at any particular moment, such as immediate needs or external circumstances during a specific period. This consideration is vital for capturing and forecasting short-term shifts in mobility patterns [1, 43] . Moreover, by formulating prompts that assume specific events of concern, this framework allows us to observe the LLM agent's responses in a variety of situations.To construct a pipeline for activity trajectory generation, we design an LLM agent with action, memory, and planning [33, 32] . Action specifies how an agent interacts with the environment and makes decisions. In LLMob, the environment contains the information collected from real-world data, and the agent acts by generating trajectories. Memory includes past actions that need to be prompted to the LLM to invoke the next action. In LLMob, memory refers to the patterns and motivations output by the agent. Planning formulates or refines a plan over past actions to handle complex tasks, with additional information optionally incorporated as feedback. In LLMob, we use planning to identify patterns and motivations, thereby handling the complex task of trajectory generation. Plan formulation, selection, reflection, and refinement [14] are employed in succession, and the agent keeps updating the action plan based on its observation [41] : The agent first formulates a set of activity plans by extracting candidate patterns from historical trajectories in the database. The agent then performs self-reflection through a self-consistency evaluation to pick the best pattern from the candidate patterns. With historical trajectories further retrieved from the database, the agent refines the identified pattern to a summarized motivation of daily activity, which is then jointly used with the identified pattern for trajectory generation. In addition to the above agentic components, we also suggest the personas of the agent, which can facilitate the LLM to simulate the diversity of real-world individuals [29] .Phase 1 of LLMob focuses on identifying activity patterns from historical data. To effectively leverage the extracted activity patterns as essential prior knowledge for the generation of daily activities, we introduce the following two steps.This step derives activity patterns based on activity trajectory data (e.g., individual check-in data). As illustrated in the left panel of Figure 2 , this scheme consists of the following aspects: For each person, we start by specifying a candidate personas to the LLM agent, providing the inspiring foundation for subsequent activity pattern generation. This approach also encourages the diversity of the generated activity patterns, as each candidate persona acts as a unique prior for the generation process (e.g., the significance of user clustering from activity trajectory data in producing meaningful distinctions has been demonstrated [24] ). Meanwhile, we perform data preprocessing to extract key information from the extensive historical data. This involves identifying usual commuting distances, pinpointing typical start and end times and locations of daily trips, and concluding the most frequently visited locations of the person. It is important to note that these pieces of information are widely recognized as critical features in mobility analysis [17] . After the preprocessing procedure, both semantic elements with historical data are combined in the prompts, requiring the LLM agent to summarize the activity patterns for this person. By doing this, we set up a streamline to effectively bridge the gap between semantic persona characteristics and concrete historical activity trajectory data, which allows for a more personalized and interpretable representation of individual activities in one day. Moreover, we propose adding candidate personas to the prompt during candidate pattern generation to promote the diversity of the results. Without loss of generality, for each person, a set of C (C = 10) candidate patterns, denoted as CP, are generated according to the historical data and C candidate personas, respectively. We provide the details of these candidate personas in Appendix C.4.This step involves assessing the consistency of the candidate patterns to identify the most plausible one. We implement a scoring mechanism to evaluate the alignment of candidate patterns with historical data. To achieve this objective, we define a scoring function to gauge each candidate pattern cp in the set CP. This function evaluates cp against two distinct sets of activity trajectories: the specific activity trajectories T i of a targeted resident i and the sampled activity trajectories from other residents T ∼i :EQUATIONwhere we design an evaluation prompt to ask the LLM to generate rating scores r t and r t ′ . Specifically, the LLM agent is prompted to assess the degree of preference for a given trajectory based on the candidate pattern. Ideally, the LLM agent should assign a higher r t for data from the targeted resident and a lower r t ′ for data from other residents. This scheme essentially identifies the self-consistent pattern: the activity pattern derived from the activity trajectory data of the target user should be consistent with the data from this person during the evaluation. We provide the pseudo-code of the algorithm for Phase 1 of LLMob in Appendix A.In Phase 2 of LLMob, we focus on the retrieval of motivation and the integration of motivation and activity patterns for individual activity trajectory generation. Since the context length is limited for the LLMs, we can not expect that the LLMs can consume all the available historical information and give plausible output. Retrieval-augmented generation has been identified as a crucial factor in boosting the performance of LLM [37] . This enhancement provides additional information that aids LLM in more effectively responding to queries. While previous studies on activity generation mainly overlook the critical factors of macro temporal information (e.g., date) or specific scenarios (e.g., harsh weather) [42] , we propose a more sophisticated activity generation which accounts for various conditions by taking advantage of the human-like intelligence of LLM. For instance, the activity trajectory at date d can be inferred given the motivation of this date and the habitual activity pattern as:EQUATIONThis generation scheme instructs the LLM agent to simulate a designated individual according to a given activity pattern, and then meticulously generate an activity trajectory in accordance with the daily motivation. To obtain insightful and reliable motivations toward different aspects of data availability and sufficiency, two retrieval schemes are proposed. Notably, we considered them as two promising directions for designing solutions to real-world applications, rather than claming which is superior. The detail of each retrieval scheme is introduced as follows: This scheme is related to the intuitive principle that an individual's motivation on any given day is influenced by her interests and priorities in preceding days [28] . Guided by this understanding, our approach harnesses the intelligence of the LLM agent to understand the behavior of daily activities and the underlying motivations. As illustrated in Figure 3 , for a specific date d for which we aim to generate the activity trajectory, we consider the activities of the past k days (k = min(7, l), where l is the maximum value such that the trajectory for date d -l can be found in the database), and prompt the LLM agent to act as an urban resident based on the pattern identified in Section 3.1 and summarize k motivations behind these activities. Using these summarized motivations, the LLM agent is further prompted to infer potential motivation for the target date d.In this scheme, we hypothesize that individuals tend to establish routines in their daily activities, guided by consistent motivations even if the specific locations may vary. For example, if someone frequently visits a burger shop on weekday mornings, this behavior might suggest a motivation for a quick breakfast. Based on this, it is plausible to predict that the same individual might choose a different fast food restaurant in the future, motivated by a similar desire for convenience and speed during their morning meal. We introduce a learning-based scheme to retrieve motivation from historical data. For each new date on which to plan activities, the only information available is the date itself. To use this clue for planning, we first formulate a relative temporal feature z dc,dp between a past date d p and the current date d c . This feature captures various aspects, such as the gap between these two dates and whether they belong to the same month. Utilizing this setting, we train a score approximator f θ (z dc,dp ) to evaluate the similarity between any two dates. Notably, due to the lack of supervised signals, we employ unsupervised learning to train f θ (•). Particularly, a learning scheme based on contrastive learning [5] is established. For each trajectory of a resident, we can scan her other trajectories and identify similar (positive) and dissimilar (negative) dates according to a predefined similarity score. This similarity score is calculated between two activity trajectories T da and T d b as:EQUATIONwhere N d is the total number of time intervals (e.g., 10 min) in one day. T da (t) indicates the tth visiting location recorded in trajectory T da . Intuitively, there should be more shared locations in the similar trajectory pair. Thereafter, the positive pair is characterized by the highest similarity score, indicative of a greater degree of resemblance between the trajectories. Conversely, the negative pairs are marked by low similarity scores, reflecting a lesser degree of commonality. After obtaining the training dataset from these positive and negative pairs, we train a model to approximate the similarity score between any two dates by contrastive learning. This procedure involves the following steps: EQUATION3. Adopt InfoNCE [25] as the contrastive loss function:L(θ) = N n=1-log e logits i k+1 j=1 e logits j n ,where N is the batch size of the samples and i indicates the index of the positive pair.Upon training a similarity score approximation mode, it can be applied to access the similarity between any given query date and historical dates. This enables us to retrieve the most similar historical data, which is prompted to the LLM agent to generate a summary of the motivations prevalent at that time. By doing so, we can extrapolate a motivation relevant to the query date, providing a basis for the LLM agent to generate a new activity trajectory. We utilize the category classification in Foursquare to determine the activity category for each location. We use 10 candidate personas (Appendix C.4) as a prior for subsequent pattern generation, which captures a diverse range of activity patterns within the data of this study. For the application to other datasets, this style of candidate patterns can be easily initialized using an LLM.Metrics. The following characteristics related to personal activity are used to examine the generation: (1) Step distance (SD) [42] : The travel distance between each consecutive decision step within a trajectory is collected. This metric evaluates the spatial pattern of an individual's activities by measuring the distance between two consecutive locations in a trajectory. (2) Step interval (SI) [42] : The time gap between each consecutive decision step within a trajectory is recorded. This metric evaluates the temporal pattern of an individual's activities by measuring the time interval between two successive locations on an individual's trajectory. (3) Daily activity routine distribution (DARD): For each decision step, a tuple (t, c) is created, where t represents the occurring time interval (e.g., from 0 to 144 in a day) and c identifies the activity category based on the location visited at that step. A histogram is then constructed to represent the distribution of the collected tuples. This feature presents the patterns of individual activities characterized by activity type and timing (e.g., activity type, time). It provides insight into how activities are distributed over space and time and reflects semantic information such as habitual behavior. (4) Spatial-temporal visits distribution (STVD): For each decision step, a tuple (t, latitude, longitude) is created, where t represents the occurring time interval (e.g., from 0 to 144 in a day) and latitude, longitude are the geographic coordinates of the location visited at that step. A histogram is subsequently built to represent the distribution of the collected tuples. This feature provides a granular perspective on the generated activities by assessing the spatial-temporal distribution of visited locations within each trajectory, including geographical coordinates and timestamps. It enables a detailed analysis of where and when activities occur.After extracting the above characteristics from both the generated and real-world trajectory data, Jensen-Shannon divergence (JSD) is employed to quantify the discrepancy between them. Lower JSD is preferred.Methods. LLMob is evaluated against: Markov-based mechanic model (MM) [27] , an LSTM-based prediction model (LSTM) [12] , two attention-based prediction models, including DeepMove [8] and STAN [22] . Within the domain of deep generative models, we select two adversarial learning frameworks, including TrajGAIL [6] and ActSTD [42] , as well as a diffusion model, DiffTraj [46] . We use the source codes of the baselines provided by their respective authors and adapt them to our setting.To achieve a balance between capability and cost efficiency, we employ GPT-3.5-turbo-0613 as the LLM core. We use "LLMob-E" to represent the proposal with the evolving-based motivation retrieval scheme, and "LLMob-L" to denote the framework that incorporates the learning-based motivation retrieval scheme (parameter settings in Appendix C.3). To validate the necessity of each module proposed, we conduct ablation studies with the following configurations: "LLMob w/o P" denotes the framework generating trajectories without using the pattern (i.e., directly summarizing motivations from past trajectories). "LLMob w/o M" denotes the framework without the motivation (i.e., directly generating trajectories with the identified pattern). "LLMob w/o SC" denotes the framework without the self-consistency evaluation (in this case, a candidate pattern is randomly picked as the identified pattern). Furthermore, "LLMob w/o P & M" represents the framework excluding both patterns and motivations. 
main results and analysis: 
 Generative Performance Validation (RQ 1, RQ 2). The performance evaluation involves analyzing generation results in three distinct settings: (1) Generating normal trajectories based on normal historical trajectories in 2019, a period unaffected by the pandemic. (2) Generating abnormal trajectories based on abnormal historical trajectories in 2020, a year marked by the pandemic. (3) Generating abnormal trajectories in 2021 (pandemic) based on normal historical trajectories in 2019.The results of these evaluations are detailed in the metrics reported in Table 2 . Through the comparison, it can be observed that although LLMob may not excel in replicating spatial features (SD) precisely, it demonstrates superior performance in handling temporal aspects (DI). When considering spatial-temporal features (DARD and STVD), LLMob's performance is also competitive. In particular, LLMob achieves the best performance on DI and DARD for all three settings and is the runner-up on STVD. Baselines like DeepMove and TrajGAIL perform the best on SD and STVD, respectively, but become much less competitive when evaluated in other aspects. We suggest that the pronounced advantage of LLMob in terms of DARD (roughly 1/2 to 1/3 JSD compared to the best of baselines) can be attributed to the LLM agent's tendency to accurately replicate the motivation behind individual activity behaviors. For instance, an agent may recognize patterns like a person's habits to have breakfast in the morning, without being restricted to a specific restaurant. This phenomenon highlights the enhanced semantic understanding capabilities of the LLM agent.Exploring Utility in Real-World Applications (RQ 3). We are interested in how LLMob can elevate the social benefits, particularly in the context of urban mobility. To this end, we propose an example of leveraging the flexibility and intelligence of LLM agents in understanding semantic information and simulating an unseen scenario. In particular, we enhance the original setup by incorporating an additional prompt to provide a context for the LLM agent, enabling it to plan activities during specific circumstances. For example, a "pandemic" prompt is as follows: Now it is the pandemic period. The government has asked residents to postpone travel and events and to telecommute as much as possible. By integrating the above prompt, we can observe the impact of external elements, such as the pandemic and the government's measures, on urban mobility and related social dynamics. We use the activity trajectory data during the pandemic (2021) as ground truth and plot the daily activity frequency in 7 categories in Figure 4 . TrajGAIL, despite delivering the best STVD in Table 2 , displays very low frequencies for all the categories, and fails to reflect the tendency of each category. In contrast, a comparison between LLMob-L and the one augmented with the pandemic prompt demonstrates the impact of external factors: there is a significant decrease in activity frequency with the pandemic prompt, which semantically discourages activities likely to spread the disease (e.g., food).Additionally, from a spatial-temporal perspective, two major activities (e.g., Arts & entertainment and Professional & other places) are selected to observe the behavior, as shown in Figures 5a and 5b . These activities are particularly insightful as they encapsulate the impact of the pandemic on the work-life balance and daily routines of residents. Specifically, with the pandemic prompt, LLMob reproduces a more realistic spatial-temporal activity pattern. This enhanced realism in the generation is attributed to the integration of prior knowledge about the pandemic's effects and governmental responses, allowing the LLM agent to behave in a manner that aligns with actual behavioral adaptations. For instance, the reduction in Arts & entertainment activities reflects the closure of venues and social distancing guidelines, while changes in Professional & other places activities indicate shifts toward remote work and the transformation of professional environments. Intuitively, prompting the LLM agent to generate activities based on various priors shows great potential in real-world applications. The utility of such a conditioned generative approach, coupled with the reliable generated results, can significantly alleviate the workload of urban managers. We suggest that this kind of workflow can simplify the analysis of urban dynamics and aid in assessing the potential impact of urban policies.Impact of Patterns. In Table 2 , by comparing LLMob with and without using patterns ("w/o P"), we observe that the identified patterns consistently enhance the trajectory generation performance. The improvement on DARD is the most significant (reducing JSD by around 50%), showcasing the use of patterns is a key factor in capturing the semantics of daily activity. We provide example patterns in Appendix D.1 to show how the habitual behaviors of individuals are recognized by patterns.Impact of Self-Consistency Evaluation. By comparing LLMob with and without self-consistency evaluation ("w/o SC") in Table 2 , we find that self-consistency is useful in all aspects, and its impact is the most significant on DARD, especially when generating abnormal trajectories from normal data, showcasing its effectiveness in processing semantics. We also observe that "w/o SC" performs even worse than "w/o P" in many cases, because in "w/o SC", a candidate pattern is randomly picked for summarizing motivations, potentially introducing inconsistency to an individual's daily activity.Impact of Motivations. We compare LLMob with and without motivations ("w/o M"). As can been seen in Table 2 , the impact of motivations is similar to that of patterns. By comparing to LLMob with both patterns and motivations removed ("w/o P & M"), we observe that these two factors collectively lead to better performance. To show the motivations and the generated trajectories, we provide examples in Appendix D.2, where consistency between them can be observed.Impact of Motivation Retrieval Strategy. We compare LLMob equipped with the two motivation retrieval strategies ("-E" and "-L"). Table 2 shows that no retrieval strategy always outperforms the other, though evolving-based retrieval wins in more cases (7 vs 5). Moreover, evolving-based retrieval is generally less sensitive to the removal of patterns or self-consistency evaluation, suggesting that resorting to the LLM to process historical trajectories is more robust than using contrastive learning.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: We provide the source codes of this study. The trajectory dataset is provided in an anonymous setting. Researchers can reproduce the results by running our source codes on this opensource anonymous dataset. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
conclusion: 
 Contributions. This study is believed to be the first personal mobility simulation empowered by LLM agents on real-world data. Our innovative framework leverages activity patterns and motivations to direct LLM agents in emulating urban residents, facilitating the generation of interpretable and effective individual activity trajectories. Extensive experimental studies based on real-world data are conducted to validate the proposed framework and demonstrate the promising capabilities of LLM agents to improve urban mobility analysis. Social Impacts. Leveraging artificial intelligence to enhance societal benefits is increasingly promising, especially with the advent of high-capacity models such as LLMs. This study explores one of the potential avenues for applications using LLMs as reliable agents to simulate specific scenarios to assess the effects of external factors, such as pandemics and government policies. The introduced framework offers a flexible approach to enhance the reliability of LLMs in simulating urban mobility.Limitations. In this study, we focused on modeling the activities of individual agents without considering interactions between them. As future work, we aim to extend this to a multi-agent scenario to capture interactions (e.g., where individuals may follow the activities of friends or family members). Given the challenges in collecting high-quality personal mobility data-many datasets lack completeness in capturing daily activities-we limited our comprehensive experimental evaluation to a single dataset. Furthermore, due to cost-efficiency considerations, only GPT-3.5 was fully evaluated. An additional analysis in a different city is provided in Appendix D.3, and Appendix D.4 includes supplementary evaluations using other LLMs. Student: typically travel to and from educational institutions at similar times.Teacher: typically travel to and from educational institutions at similar times.Office worker: have a fixed morning and evening commute, often heading to office districts or commercial centers.Visitor: tend to travel throughout the day, often visit attractions, dining areas, and shopping districts.Night shift worker: might travel outside of standard business hours, often later in the evening or at night.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We provide the source codes of this study. The trajectory dataset is not provided to its intellectual property. Please refer to the supplementary material for our source codes. Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: We use the GPT-3.5 API but not release any models. Due to the intellectual property, the dataset is not released. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Service industry worker: tend to travel throughout the day, often visit attractions, dining areas, and shopping districts.Public service official: often work in shifts, leading to varied travel times throughout the day and night.Fitness enthusiast: often travel early in the morning, in the evening, or on weekends to fitness centers or parks.Retail employee: travel patterns might include shifts that start late in the morning and end in the evening.You are a student in this urban neighborhood. You typically travel to and from educational institutions at similar times.Your daily routine as a student in an urban neighborhood revolves around traveling to and from educational institutions. On weekdays, you cover a distance of over 10 kilometers, starting your daily trip at 12:00 and concluding it at 20:20. At the beginning of the day, you usually visit Park#2457 and then head to Grocery Store#648 before returning home. Weekends follow a similar pattern, with you traveling over 10 kilometers a day, but starting your daily trip at 13:40 and ending it at 19:20. Like on weekdays, you begin by visiting Park#2457 and then proceed to Grocery Store#648 before heading back home. Throughout the week, you have specific places You visit at fixed times, such as Grocery Store#648 at 20:00, Coffee Shop#571 at 09:30, Park#2457 at 08:30, Irish Pub#21 at 20:00, and Bookstore#313 at 16:00. Additionally, there are other locations You occasionally visit, including Chinese Restaurant#168, Supermarket#802, Park#4719, Tea Room#530, Bookstore#336, Pastry Shop#240, Park#2898, Discount Store#807, and Electronics Store#530. By visiting educational institutions at similar times, you ensure punctuality and consistency in your studies. Additionally, your visits to the park and various stores serve as a way to relax, replenish supplies, and indulge in leisure activities. Overall, your daily routine is structured to fulfill both your educational and personal requirements efficiently.You are a teacher in this urban neighborhood. You typically travel to and from educational institutions at similar times.Your daily routine as a teacher in an urban neighborhood revolves around your regular travel to and from educational institutions. On weekdays, you cover a distance of over 60 kilometers a day, starting your journey at 11:50 in the morning and concluding it at 17:50 in the evening. At the beginning of the day, you usually visit Rest Area#1533 before heading to the Housing Development#101, where you spend a significant amount of time before returning home. During weekends, your travel distance reduces to 50 kilometers a day, with your daily trip commencing at 11:20 and ending at 18:00. On weekends, you follow a different routine, starting your day by visiting Shopping Mall#1262 and then proceeding to Motorcycle Shop#149 before heading back home. Additionally, there are certain locations You occasionally visit, such as Convention Center#101, Food Court#559, Motorcycle Shop#354, and Sports Bar#56, among others. Your daily routine is motivated by the need to fulfill your teaching responsibilities and ensure that you are present at the educational institutions You serve. The specific locations You visit, whether it's the Rest Area#1533 or Shopping Mall#1262, play a role in providing you with necessary resources, relaxation, or opportunities to engage in personal interests. Overall, your routine is structured to maintain a balance between professional commitments and personal needs.r relevant places, you can effectively serve the needs of the urban neighborhood and contribute to its smooth functioning.You are a office worker in this urban neighborhood. You have a fixed morning and evening commute, often heading to office districts or commercial centers.Your daily routine as an office worker in an urban neighborhood is quite structured. On weekdays, you travel over 30 kilometers a day, starting your daily trip at 06:50 and ending it at 20:20. Your routine usually begins with a visit to Platform#212, followed by heading to Soba Restaurant#955 before returning home. You also have certain places You visit at specific times, such as Hospital#1255 at 10:30, Platform#212 at 06:30, Soba Restaurant#955 at 22:30, Public Bathroom#76 at 08:30, and Platform#1068 at 05:00. During weekends, your daily travel distance increases to over 40 kilometers, and you start your trip at 07:00, ending it at 20:10. However, the pattern remains the same, starting with a visit to Platform#212 and then going to Soba Restaurant#955 before returning home.You are a visitor in this urban neighborhood. You tend to travel throughout the day, often visit attractions, dining areas, and shopping districts.Your daily routine as a visitor in this urban neighborhood involves traveling extensively and exploring various attractions, dining areas, and shopping districts. On weekdays, you cover a distance of over 40 kilometers, starting your day at 11:00 and concluding it at 17:10. You have a consistent pattern of visiting Ramen Restaurant#4841 at the beginning of the day and then heading to Town#373 before returning home. Weekends are slightly different, with a shorter distance of around 30 kilometers covered. You begin your day at 14:00 and end it at 16:50. During weekends, you start by visiting Ramen Restaurant#3773 and then proceed to Town#373 before heading back. There are certain locations that you frequently visit, such as Arcade#929 at 13:30, Bowling Alley#306 at 11:00, Arcade#408 at 12:00, and Grocery Store#2094 at 12:30. Additionally, you sometimes visit other places like Electronics Store#562, Comic Shop#5, Video Game Store#8, and many others. Your routine is motivated by your desire to explore and experience the various attractions, cuisines, and shopping opportunities available in this urban neighborhood. You find joy in discovering new places, trying different foods, and immersing yourself in the vibrant atmosphere of this bustling area.You are a night shift worker in this urban neighborhood. You travel to work in the evening and return home in the early morning.Your daily routine as a night shift worker in an urban neighborhood revolves around traveling a considerable distance. On weekdays, you cover over 70 kilometers a day, starting your daily trip at 09:20 and concluding it at 22:30. The routine begins with a visit to Toll Booth#194, followed by a stop at Supermarket#3823 before heading home. During weekends, the pattern remains the same, with the only difference being that you commence your journey at 10:30 and finish at 22:20. On these days, you visit Toll Booth#812 first and then head to Cafe#962 before returning home. In addition to these regular stops, there are occasional visits to various locations such as Record Shop#81, Lake#600, and Factory#495, among others. The motivation behind this routine is to ensure that you are well-prepared for your night shift, starting the day by taking care of essential tasks like stopping at toll booths and getting groceries. These routines help you maintain a sense of order and efficiency in your daily life, ensuring that you are ready for work and able to relax and enjoy your free time.You are a remote worker in this urban neighborhood. You may have non-standard travel patterns, often visit coworking spaces or cafes at various times.Your daily routine as a remote worker in an urban neighborhood involves traveling a considerable distance, both during weekdays and weekends. On weekdays, you typically travel over 40 kilometers a day, starting your daily trip at 14:00 and ending it at 19:20. Your routine usually begins with a visit to Convention Center#101, followed by a stop at Supermarket#1593 before returning home. During weekends, your travel distance is slightly less, around 20 kilometers a day, and you begin your daily trip at 11:50, concluding it at 16:30. On weekends, you usually start by visiting Discount Store#884 and then head to Supermarket#1689 before returning home. Additionally, you have specific places You visit at certain times, including Supermarket#1593 at 17:00, Hobby Shop#516 at 13:30, Shopping Mall#1262 at 15:00, Hobby Shop#168 at 14:00, and Exhibit#461 at 11:30. Occasionally, you also visit other locations such as Shopping Mall#1073, Bookstore#14, Shrine#2783, and many more. Your motivation for this routine is to have a flexible work environment, utilizing coworking spaces and cafes, while also fulfilling your daily needs and exploring different places within the urban neighborhood.You are a service industry worker in this urban neighborhood. You might travel outside of standard business hours, often later in the evening or at night.Your daily routine as a service industry worker in an urban neighborhood is quite busy and revolves around your work schedule. On weekdays, you travel over 10 kilometers a day, starting your daily trip at 07:20 and ending it at 20:40. The day usually begins with a visit to Historic Site#2176, followed by a stop at Convenience Store#3385 before returning home. During weekends, your travel distance decreases to 0 kilometers a day, starting your daily trip at 09:20 and ending it at 20:10. Similar to weekdays, you start your day by visiting Historic Site#2176 and then go to Public Art#99 before returning home. Additionally, there are certain locations You sometimes visit, including various convenience stores, restaurants, shopping malls, and other establishments. Your motivation for this routine is primarily driven by your work commitments and the need to fulfill your responsibilities as a service industry worker. It is essential for you to visit specific places, such as convenience stores and historic sites, to ensure You have the necessary supplies and maintain a well-rounded understanding of the neighborhood. Overall, this routine enables you to efficiently navigate your urban neighborhood and fulfill your professional obligations.You are a public service official in this urban neighborhood. You often work in shifts, leading to varied travel times throughout the day and night.Your daily routine as a public service official in an urban neighborhood revolves around your shifts, which result in different travel times. On weekdays, you typically cover over 60 kilometers a day, starting your daily trip at 12:50 and concluding it at 19:00. At the beginning of the day, you make it a point to visit Convenience Store#3042, and before heading home, you stop by Convenience Store#3702. During the weekends, you travel around 50 kilometers daily, commencing your journey at 10:00 and wrapping it up at 18:30. On weekends, your routine involves visiting Platform#511 in the morning and then heading to Platform#670 before returning home. Additionally, there are a few locations You occasionally visit, such as Platform#1135, Home Service#244, and Convenience Store#6014, among others. The motivation behind your daily routine is to ensure that you cover the necessary ground, making essential stops at various locations to fulfill your duties as a public service official. By visiting convenience stores, platforms, home services, and other relevant places, you can effectively serve the needs of the urban neighborhood and contribute to its smooth functioning.You are a fitness enthusiast in this urban neighborhood. You often travel early in the morning, in the evening, or on weekends to fitness centers or parks. Every weekday, you travel over 70 kilometers a day, starting your daily trip at 10:30 and ending it at 19:50. Your routine begins with a visit to Toll Booth#34, where you kickstart your day. After that, you head to Recreation Center#18 to engage in your fitness activities before returning home. On weekends, your daily travel distance is slightly less, around 60 kilometers. You start your trips at 12:40 and end them at 21:00. The weekend routine starts with a visit to Convenience Store#5940, where you grab some essentials for the day. Then, you head to Convenience Store#8965 before finally returning home. Throughout the week, you also occasionally visit other locations such as Tunnel#1307, Event Space#104, Shopping Mall#217 and #399, and various toll booths. Your motivation for this daily routine is your passion for fitness and maintaining a healthy lifestyle. You prioritize visiting fitness centers or parks during your trips to ensure that you have dedicated time for exercise. Additionally, you make sure to visit convenience stores for any necessary supplies and toll booths for smooth travel. Overall, your routine revolves around staying active, exploring different locations, and ensuring a well-rounded fitness experience.You are a retail employee in this urban neighborhood. Your travel patterns might include shifts that start late in the morning and end in the evening. Every weekday, you embark on a daily journey that spans over 50 kilometers. Your routine begins at 09:30, and you conclude your travels at 17:00. To kickstart your day, you always make a point to visit Soba Restaurant#2105, relishing in their delicious offerings. Before heading home, you make a stop at Indian Restaurant#885, savoring their delectable cuisine. On weekends, your daily travel distance decreases slightly to 40 kilometers. You commence your excursions at 09:10 and wrap them up at 17:20. Your first stop on weekends is Hot Spring#205, where you indulge in relaxation and rejuvenation. Before returning home, you make a detour to Department Store#399, exploring the vast array of products they offer. Additionally, there are several other locations You occasionally visit, such as pharmacies, history museums, shrines, shopping malls, and more. Your motivation for this routine stems from your desire for variety and exploration. By visiting different establishments and places, you are able to experience diverse cuisines, immerse yourself in local culture, and find moments of serenity amidst the bustling city life.• Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in the appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 


Decision: This paper introduces LLMob, a novel approach using Large Language Models (LLMs) integrated into an agent framework for flexible and effective personal mobility generation
The method requires insightful generation of structured prompts. It is brining the idea of agents systems to personal mobility and looks at multiple problems: including aligning LLMs with the urban data in the real world, developing strategies with reliable activities.
The empirical performance is limited, since it is on a biased and limited dataset. However the reviewer acknowledges that availability of such data is not easy, and hence showcasing this new problem may need creation and curation of the data itself. There are a number of concerns, mainly centered on clarity of writing and clarity in terms of empirical superiority of the method. The metrics used are nonintuitive and since there is no single methods that dominates, it is not clear what the value of the presented method is.  However the rebuttal does a decent job of clarifications and additional experiments such as evaluation with Llama. 

Furthermore there are ethical concerns on the privacy of the data. The authors confirmed that the data is obtained through publicly available APIs from Foursquare and Twitter, and hence follow their guidelines. The authors should follow through with the plan of additional steps to anonymize the output location depending on the specific requirements of the application.  Given the over all contributions, the meta-reviewers finds that the paper has value of the subcommunity of ML studying mobility with LLMs. The authors are however requested to make major changes improvements to the manuscript based on the discussion feedback to improve writing, include new experiments, acknowledge limitations and suggest solutions for the data anonymity.

Reviews: review: paper_summary: paper_summary: This paper presents a prompt engineering framework for generating synthetic human trajectories using LLMs. The framework is guided, in an overall manner, by the observation that human movement is affected by habitual activity patterns and motivations.  In this way, the framework has two phases for considering these aspects. In phase 2, two varying strategies are proposed and compared. All in all, this is an interesting piece of application work of LLMs, in prompt engineering.
strength_weakness: strength_weakness: 1)	There are some parts in the text that are difficult to decipher – I wouldn’t say the paper is easy to follow.
2)	It is unclear which method actually yields the best performance across different metrics
3)	The generalizability is unclear. Can this method be used in other cities around the globe?
questions: questions: 1)	Is this method zero-shot?
2)	How is the Td in 3.2 (habitual activity pattern) used subsequently?
3)	What’s the relation between 3.2.1 evolving based motivation retrieval and 3.2.2 learning-based motivation retrieval? 
4)	Learning-based motivation retrieval appears to be dependent on embeddings obtained through a contrastive learning process. In what way the embeddings are actually used?
5)	What’s the rationale behind the two retrieval strategies? If one is superior, why the other is being discussed?
limitations: limitations: See weaknesses
paper_summary: paper_summary: This paper introduces LLMob, a framework for personal mobility generation using a large language model. The framework aims to leverage urban activity patterns for emulating urban residents, facilitating the human mobility trajectory generation. Using the Tokyo personal activity dataset, the effectiveness of the proposed framework are validated.
strength_weakness: strength_weakness: - The proposed framework has only been validated in GPT-3.5. It remains unclear whether the performance would vary with different backbone models, such as Llama-2.
- The constructed Tokyo dataset statistic is not very clear in this paper, which makes it difficult for the reader to evaluate the true contribution of this paper.
- The difference between this work and existing work (such as [1]) is not well-discussed in the related work section, although the author claims why these methods cannot be easily adopted as the baselines.

[1] Shao, Chenyang, et al. "Beyond Imitation: Generating Human Mobility from Context-aware Reasoning with Large Language Models." arXiv preprint arXiv:2402.09836 (2024).
questions: questions: - In Line 250 - Line 253, 100 users are chosen and 10 candidate personas are used for subsequent pattern generation. Why did the author select such a setting, I don't see a clear illustration of these settings in the paper.
- In Line 46 and Line 45, the author mentioned "unseen tasks" and "unseen scenarios". In my perspective, the personal mobility generation task is well-formulated in this paper, what does "unseen" mean here for those data-driven methods?
limitations: limitations: In this paper, all experiments are conducted on a single collected dataset (perhaps thousands of trajectories) from Tokyo, which makes this paper not comprehensive enough. Extending the proposed method to more cities helps improve paper quality.
paper_summary: paper_summary: This paper proposes an LLM-based agent framework for generating personal mobility. It tries to address several problems in the domain of personal mobility generation, including aligning LLMs with the urban data in the real world, developing strategies with reliable activities, and exploring further applications in this field. For generating activities of personal mobility (i.e., trajectories), it proposes a framework of LLM-based agent with an action module, memory module, and planning module, which of all these modules can improve the alignment of the results of generation and real-world data. The paper provides several experiments to verify the effectiveness and alignment of the proposed methods and explores further enhancement.
strength_weakness: strength_weakness: 1. What about other LLMs performing as the core, besides GPT-3.5-turbo? I think GPT-4/GPT-4o can be utilized to generate data of higher quality.
2. For the method of Learning-based Motivation Retrieval, I'm a bit concerned about the calculation of similarity among dates. Although the locations are greatly important to identify, there seems to be too much information lost from their initial motivations. Could you try just using cosine similarities on the semantic embeddings of contents with language models? Or maybe using LLMs with prompting to get the score of similarity?
3. I think the presentation can be improved. A preliminary can be added, and the overview of LLMob can be more clear at the start of section 3.
questions: questions: Please see the above "Weakness". I'm willing to improve my rating if the authors can address my concerns.
limitations: limitations: The authors have discussed the limitations in the last section.
paper_summary: paper_summary: The paper proposes an LLM-based personal activity generation by utilizing existing OpenAI LLM through API, named as LLMob. The problem that authors are time and important research topic in the area of human mobility simulation. While existing generative solely trained on the given dataset, LLM agent has already trained internet-wide dataset that allows it to have possibility of generating human-like trajectories. The LLMob generates activity trajectories in two stages: the first stage identifies (learns) activity patterns with respect to different personas introduced by the authors, and in the second stage a synthetic version of the activity sequences generated by LLM agent. The experiments are performed on an activity dataset, and performance is benchmarked by multiple generative models.
strength_weakness: strength_weakness: These are the set of  weaknesses of the paper from my perspective:
1-) The authors performed experiments on one dataset. It might be interesting to examine the performance with dataset from a different region. 
2-) The model solely relies on the pre-trained LLM models. While the LLM capabilities and power are increasing, which is good for this work, separate fine-tuning might be required for concrete human mobility modeling. 
3-) The motivation retrieval part is not clear, especially the learning-based motivation retrieval.
questions: questions: While Phase 1 and Phase 2 make sense. I have concerns about how much information was carried from Phase 1 to Phase 2. The following are the questions that can help me to understand the details:
 
1-) Did you utilize the whole dataset in Phase 1? 
2-) How are the trajectories and personas sampled in Phase 2?
3-) How do the generated activities match with the input trajectories? Does LLM memorize exact activity sequences?
4-) LLMs are known for hallucination. Within generated activities, the LLM agent generates an activity name and location. How accurate is the LLM agent for the generated activities? For instance, when an activity says Japanese Restaurant (35.369, 140.306), does the GPS location have a Japanese Restaurant? I expect the generated trajectories to require extensive post-processing. 
5-) Have you tried your model on a different dataset? Experiments are limited to one dataset, which limits generalizability.
limitations: limitations: I found two limitations in this paper and would love to hear authors responses on these:

1-) While LLMs are generally powerful, they often struggle to generalize in domain-specific applications. For a robust human mobility simulation, LLM agents might require a fine-tuning strategy to achieve more generalizable performance. 

2-) The generated locations in the study are limited to those specified in the prompt (If they do not hallucinate). Since the model also takes into account the locations a person is likely to visit, as illustrated in Figure 2, the degree of synthetic realism in the generated trajectories raises questions. 

3-) This approach also does not fully address privacy concerns associated with mobility trajectories as mentioned in the introduction, the underlying patterns and personal data might still be exposed.

Comments: title: Summary of Reviews and Rebuttals
Re: Thanks
Response to ethics review
Thank you for the comments
Thanks
Re: Thanks
Thanks
Re: Official Comment by Reviewer c5UM
Response to new comments from reviewer xLxx (2/2)
Response to new comments from reviewer xLxx (1/2)
Response to The Author
Response to Reviewer's Engagement
Response to new comments from reviewer xLxx
Response to The Author
Rebuttal by Authors
comment: **Table Reviewer Rating Summarization.**

| Score/Reviewer | Reviewer abCw | Reviewer xLxx                                         | Reviewer 7xtG | Reviewer c5UM | Average    |
| -------------- | ------------- | ----------------------------------------------------- | ------------- | ------------- | --- |
| Rating         | 6             | 4  | 5             | 6             |  5.25    |
| Confidence    | 4             | 4                                                     | 3             | 4             |  3.75   |

From the table, it is clear that **three reviewers** have been **very positive** towards our paper.

Our work is the **first work** to **employ LLM agent for semantic human mobility generation**, which is a **very important and pioneering attempt in the field of LLM for computational social science**. Hope we could get your credit and kind support. Thanks a lot!
Following our previous post, in which we tried to address the most recent concerns of the reviewer, we sincerely appreciate the reviewer's dedicated efforts in evaluating our paper and engaging in the discussion. Throughout the discussions, we have tried our best to address the raised questions, particularly regarding the dataset details (construction & statistics) and the inclusion of additional experiments. We find these discussions highly constructive for improving the quality of our paper. 

Regarding the identified limitations, we acknowledge that expanding the number of users and candidate personas could potentially enhance the data's sufficiency and diversity. However, we wish to emphasize that our current selection of 100 users and 10 personas has already yielded **promising outcomes**, as demonstrated in our results. It is important to note that increasing these numbers would lead to **higher token consumption and computational costs**. This limitation is not unique to our study but is also **observed in many notable existing works** that utilize GPT 3.5/4 APIs, including the seminal work of generative agents [1], which involved only 25 agents.

[1] Park J S, O'Brien J, Cai C J, et al. Generative agents: Interactive simulacra of human behavior. Proceedings of the 36th annual ACM symposium on user interface software and technology. 2023: 1-22.

We are committed to addressing all concerns highlighted by the reviewer in the next version of our paper, incorporating all details from this discussion. We hope that these efforts meet the reviewer's expectations and merit consideration for a higher rating.
Thank you for your valuable comments. We fully understand and acknowledge the importance of privacy issues and would like to address your concerns as follows:

> **Comment 1**: *The paper uses a dataset "obtained through Twitter and Foursquare APIs and covers the data from January 2019 to December 2022". This is potentially personally identifiable information. It is also not clear whether the location information is collected with the consent of the individuals who have been tracked. Considering that movement patterns are highly individual it is also possible that individuals can be de-identified from such data. Details or a discussion about the privacy concerns of using such a dataset, as well as the measures put in place to reduce such risks are missing. I would recommend that the authors describe further the data and the measures they've taken to ensure the privacy of the individual whose data they are using.*

**Answer:**  We would like to clarify that all the data used in our study is indeed anonymous. First of all, the data obtained through the Twitter and Foursquare APIs was already anonymized to remove any personally identifiable information before analysis. Note that the Foursquare data we collect strictly obeys the privacy policy in Foursquare, and the Twitter data we collect is obtained through Twitter Inc.'s Academic Research Product Track, ensuring compliance with their guidelines. The data collection and preprocessing is similar to the one in study [1]. When we construct the input of the model, the userid is a random number and the locations with GPS coordinates (e.g., Convenience Store (35.697, 139.416) ) are replaced with a location name and a unique number (e.g., Convenience Store # 1). We will include a more detailed description of these processes in the appendix of the paper next-version.

[1] Yang, D., Zhang, D., & Qu, B. (2016). Participatory cultural mapping based on collective behavior data in location-based social networks. ACM Transactions on Intelligent Systems and Technology (TIST), 7(3), 1-23

> **Comment 2**: *Furthermore, the paper explores tools that ease the analysis of people's movement. While the paper's "Social impacts" paragraph stresses the social utility of such tools, the paper fails to address or discuss their potential negative impacts. For example, such a tool can help track, de-identify, and target individuals at scale, raising concerns regarding their privacy and safety. I would recommend that the authors do a broader and more nuanced assessment of the potential social impacts of their work.*

**Answer:** We agree with the need to provide a more detailed discussion on the potential negative impacts associated with real-world applications of our work. In this paper, we introduce one promising application of our model, which is used to simulate the urban mobility dynamic under the pandemic. In this demonstration, we only consider aggregated movement data within an area of interest, therefore individual privacy is well maintained. We also would like to point out that, as the input data is fully anonymized, there is no possibility of inferring or targeting any specific individual from the model's output. To further ensure personal privacy in other possible applications, we suggest that we further anonymize the output location depending on the specific requirements of the application.
Thank you for your time and the valuable comments on this paper. We would like to address your concerns as much as possible:

> **Concern 1**:  *The dataset construction details, which are very important for the researcher to follow this work.*

**Answer:** We agree with the necessity of explaining the dataset construction. In this paper, the data format has been given in Table 2, and dataset statistics has been reported in the attached pdf file in the above "more detailed information about the experimental data". 

We obtained the dataset through Twitter (now X)'s Academic Research Product Track. The construction of the dataset is reported as follows:

1. **Filtering Incomplete Data**: Users with missing check-ins for a specific year were filtered out.

2. **Excluding Non-Japan Check-ins**: Check-ins that occurred outside of Japan were removed.

3. **Inferring Prefecture from GPS Coordinates**: Prefectures were inferred based on the latitude and longitude data of check-ins.

4. **Assigning Prefecture**: Users were assigned to a prefecture based on their primary check-in location; e.g., users whose top check-in location is Tokyo were categorized as belonging to Tokyo.

5. **Removing Sudden-Move Check-ins**: Check-ins showing abrupt, unrealistic location changes, such as from Tokyo to the Osaka within a very short time frame, were deleted to remove data drift (i.e., fake check-ins), following the criteria proposed by [1].

[1] Yang, D., Zhang, D., & Qu, B. (2016). Participatory cultural mapping based on collective behavior data in location-based social networks. ACM Transactions on Intelligent Systems and Technology (TIST), 7(3), 1-23

6. **Anonymizing Data**: Real user IDs and geographic location names were anonymized. Only category information of Geographic location was kept, and latitude and longitude coordinates were converted into IDs before being input into the model.

In the next version of the paper, we will cover the construction process in the appendix, including the raw data collection and the preprocessing. We will also provide open data demos and codes related to this research. 

> **Concern 2**:  *the Lack of statistically rigorous reason behind some settings, e.g., if 100 users and 10 candidates*

**Answer:** We would like to emphasize that the chosen settings, such as 100 users and 10 candidates, were selected based on practical considerations and the specific context of our study. These settings were chosen to manage a balance between computational efficiency and the ability to demonstrate the effectiveness of our model. The effectiveness has been shown in the experimental section, and the efficiency result has been reported in the above discussion.

> **Concern 3**: *The validation size is the obvious limitation of this paper.*

**Answer:** We agree that having a larger validation size would be beneficial in demonstrating the model's performance. However, due to constraints including the data availability and computational resources, we had to make practical decisions regarding the validation size in this research. We believe that our chosen setting is reasonable, which is partly supported by the similar evaluation setting taken within the 25-LLM-agent community constructed in [1].

[1] Park J S, O'Brien J, Cai C J, et al. Generative agents: Interactive simulacra of human behavior. Proceedings of the 36th annual ACM symposium on user interface software and technology. 2023: 1-22.
After carefully reading the author's response, partial concerns were addressed. However, some important parts are still not clear: (1) The dataset construction details, which are very important for the researcher to follow this work; (2) the Lack of statistically rigorous reason behind some settings, e.g., if 100 users and 10 candidates ); (3) The validation size is the obvious limitation of this paper. 

Hope these comments and suggestions could help the author further improve their paper quality.


Best,
Thank you very much for kindly taking the time to respond to our rebuttal! We also greatly appreciate your valuable comments and suggestions. They will be reflected in the revised paper.
I would like to thank the authors for the response, and I would like to raise my score.
Thank you very much for kindly taking the time to respond to our rebuttal. We greatly appreciate your valuable comments and suggestions. They will be reflected in the revised paper.
I would like to thank the authors for their detailed rebuttal. However, I will be maintaining my original score.
> **Other Weakness 1**: *The latency of the cost of invoking gpt3.5 API is not reported in this paper, which makes the efficiency of such an LLM-based framework remain a potential issue.*

For GPT-3.5 API, the average generation time per trajectory for the results reported in Table 1 is 46 seconds.

We would like to report this in the next version of the paper. Moreover, if users seek better efficiency, we suggest using open models such as Llama 3-8B, seeing its promising results as well.

> **Other Weakness 2**: *The details of dataset construction (section 4.1) are still not clear. For an LLM-based application, it is very important to clearly present these details, and this will make it easy for other researchers to follow.*

For researchers who are interested in our work, we will provide such details in the open-source repo.
Thanks for the constructive discussion. We would like to respond as follows:

> **Weakness 3**: *Although the authors claim the difference in their framework (e.g., it is a data-driven framework and features a consistency evaluation scheme), they didn't explain the advantage behind this. The difference means for what aspect makes this work different from existing work and what its potential contribution is, instead of just explain the difference in module-desgin. Unfortunately, this suggestion appears to have been disregarded by the authors.*

In the previous response, we summarized two significant differences compared to the work mentioned by the reviewer (MobiGeaR): **data-driven** framework and **data alignment** scheme. 

1. A data-driven model has the following advantage: it can continuously learn and improve from new data. As more data becomes available, the model's performance can be enhanced, making it adaptive to evolving patterns and trends. Specifically, we demonstrated that our model can automatically adjust to changes in mobility patterns caused by events like the pandemic. In contrast, the mechanistic gravity model used in MobiGeaR is an analytical model, which does not have the capability of learning from and adapting to data. This difference becomes crucial in real-world applications where urban mobility patterns are constantly changing due to factors like urban development, policy changes, or societal shifts.

2. The data alignment scheme has the advantage of ensuring the self-consistency of the identified pattern. The importance of ensuring self-consistency in using LLMs has been investigated in previous studies such as [1]. We did not find such a self-consistency mechanism in MobiGeaR. This feature is of importance for the fidelity of simulated trajectories to actual human behavior is paramount.

[1] Wang X, Wei J, Schuurmans D, et al. Self-consistency improves chain of thought reasoning in language models[J]. arXiv preprint arXiv:2203.11171, 2022.

To reflect the reviewer's concern, we would like to include a discussion on the differences from MobiGeaR in the next version of the paper.

> **Question 1**: *What I want to know is the motivation or reason behind this setting, e.g., if 100 users and 10 candidates is sufficiently representative of this problem. The author should present this but they didn't.*

Increasing the number of users and candidate personas improves the sufficiency and diversity of data. On the downside, this compromises the efficiency. We choose such numbers to balance between the two factors. We found that the choice of 100 users and 10 candidate personas is good enough for promising results. 

In the next version, we would like to include the above explanation.

> **Question 2**: *What I want to know is the definition of "unseen" for previous data-driven methods. It seems that the author explains the "unseen" for LLMs and ignores this. Is there a possibility that several traditional methods can deal with "unseen" tasks? Overall, I think the current explanation is not well-illustrative.*

By the meaning of "unseen", we meant that the data or scenarios are unseen to the model. 

For traditional mobility generation methods, we do not think they can handle unseen tasks, because "unseen" refers to data points or patterns that fall outside the distribution of the training data. These methods often struggle with extrapolation to out-of-distribution scenarios. In contrast, our LLM-based approach leverages semantic understanding and general knowledge to reason about the scenarios on which it has not been trained. 

In addition, such adaptation to unseen scenarios is easy in LLMs by using only a prompt.

> **Limitation**: *We the authors demonstrate the proposed framework on Osaka, but the validation part is still not comprehensive enough due to limited sample sizes (only thousands of trajectories here in Tokyo). Regarding the supplementary experiments on GPT-3.5, GPT-4, and Llama 3-8B, the author seems to not explain which setting they use. Is it conducted on the (Normal Trajectory, Normal Data) setting?*

We used the (Normal Trajectory, Normal Data) setting. Due to the limited time during the rebuttal period, we cannot cover all the settings. Nonetheless, we would like to report a comprehensive evaluation of the Osaka dataset in the next version of the paper.
- **About weakness 3: the difference between this work and existing work**

Although the authors claim the difference in their framework (e.g., it is a data-driven framework and features a consistency evaluation scheme), they didn't explain the advantage behind this. The difference means for what aspect makes this work different from existing work and what its potential contribution is, instead of just explain the difference in module-desgin. Unfortunately, this suggestion appears to have been disregarded by the authors.

- **About question 1**

What I want to know is the motivation or reason behind this setting, e.g., if 100 users and 10 candidates is sufficiently representative of this problem. The author should present this but they didn't.

- **About question 2: "unseen" mean here for those data-driven methods?**

What I want to know is the definition of "unseen" for previous data-driven methods. It seems that the author explains the "unseen" for LLMs and ignores this. Is there a possibility that several traditional methods can deal with "unseen" tasks? Overall, I think the current explanation is not well-illustrative.

- **About limitations on experimental setting**

We the authors demonstrate the proposed framework on Osaka, but the validation part is still not comprehensive enough due to limited sample sizes (only thousands of trajectories here in Tokyo). Regarding the supplementary experiments on GPT-3.5,  GPT-4, and Llama 3-8B, the author seems to not explain which setting they use. Is it conducted on the (Normal Trajectory, Normal Data) setting? 

- **Other weakness**
1. The latency and the cost of invoking gpt3.5 API is not reported in this paper, which makes the efficiency of such an LLM-based framework remain a potential issue.
2. The details of dataset construction (section 4.1) are still not clear. For an LLM-based application, it is very important to clearly present these details, and this will make it easy for other researchers to follow.
Thank you very much for kindly taking the time to respond to our rebuttal! We also greatly appreciate your valuable comments and suggestions. They will be reflected in the revised paper.
Thanks for the rebuttal by the authors. I would like to raise my score to 5.
Thanks for acknowledging our response and comments. We would like to highlight our arguments:

1. Regarding Weakness 3, from the review comments, we did not find the reviewer requiring an explanation on whether previous methods can be easily adopted as baselines. In addition, for the paper mentioned by the reviewer, its source code and prompt were **not available** (the repo was expired when we accessed it, and the paper did not mention what LLM was used) at the time of our submission to NeurIPS. 

2. We are sorry that the reviewer is still confused with Question 1.

3. For Question 2, our claim in the introduction is not about the experimental setting. It is essentially a claim on the advantages of using LLMs in real-world applications.
Thanks for the responses, which partially demonstrate the effectiveness of the proposed method on other LLM backbones and datasets. However, the author fails to answer if previous methods cannot be easily adopted as the baselines (mentioned in weakness 3), and fails to provide more well-illustrative reasons for several settings in this paper (mentioned in questions 1 and question 2, the author just explains this again, but didn't provide a reasonable explanation). Considering this paper also has limitations on experimental settings, which still need time and effort to refine, I will keep my score, and I hope these comments could help the author further improve their paper quality.

Best,
> **Limitation 1** *While LLMs are generally powerful, they often struggle to generalize in domain-specific applications. For a robust human mobility simulation, LLM agents might require a fine-tuning strategy to achieve more generalizable performance.*

**Answer**: We agree that fine-tuning the LLM for a specific domain may deliver better results. However, this needs careful design of instruction-tuning data, and the tuning may incur significant training costs, considering that we are using the GPT API. Indeed, we have tried fine-tuning GPT-3.5. After comparison, we chose the current approach for a better trade-off.

> **Limitation 2** *The generated locations in the study are limited to those specified in the prompt (If they do not hallucinate). Since the model also takes into account the locations a person is likely to visit, as illustrated in Figure 2, the degree of synthetic realism in the generated trajectories raises questions.*

**Answer**: We appreciate the reviewer's observation. Our approach to specifying likely locations is grounded in research by [1], which shows that individuals typically have characteristic geographical ranges for daily activities. 

[1] Alessandretti et al. The scales of human mobility. Nature, 2020, 587(7834): 402-407.

However, our method does not limit generated locations to those specified. The LLM can reason beyond these locations, potentially introducing new, plausible ones based on the context and motivation. This balances realistic patterns with diversity.

The use of location priors enhances generation efficiency and grounds outputs in realistic spatial contexts. Our evaluations, comparing generated trajectories to real-world data, demonstrate that our method produces patterns closely matching real-world data across various metrics.

In future work, we plan to explore dynamically expanding the set of potential locations based on evolving trajectory contexts, further enhancing realism and flexibility.

> **Limitation 3** *This approach also does not fully address privacy concerns associated with mobility trajectories as mentioned in the introduction, the underlying patterns and personal data might still be exposed.*

**Answer:** We suppose that the reviewer's concern comes from the data exposed to the OpenAI. 

To address this concern, we conducted experiments using Llama 3-8B on a local GPU to ensure data security. The results are reported as follows (we also tested GPT-4o-mini):

| Model          | SD | SI | DARD | STVD |
|----------------|----------|-----------|--------|--|
| LLMob-L (GPT-3.5-turbo)      |   0.049   |  0.054    |   0.126  |   0.570 | 
| LLMob-L (GPT-4o-mini)      |   0.049   |  0.055    |   0.141  |   0.577 | 
|LLMob-L (Llama 3-8B)      |   0.054   |    0.063  |  0.119   |  0.566  | 
| LLMob-E (GPT-3.5-turbo)      |   0.053   |  0.046    |   0.125  |   0.559 | 
| LLMob-E (GPT-4o-mini)      |   0.041   |   0.053    |   0.211   |   0.531 | 
| LLMob-E (Llama 3-8B)      |  0.054   |   0.059   |  0.122    |   0.561  | 

We observed competitive performance of our framework when other LLMs were used. In particular, Llama 3-8B is overall the best when spatial and temporal factors are evaluated together (DARD and STVD). Such results demonstrate the robustness of our framework across different LLMs. On the downside, there are a few cases when Llama 3-8B generated locations that do not exist. Such hallucination was not observed on GPT-3.5-turbo and GPT-4o-mini. Nonetheless, we expect that the promising results of this open LLM can inspire further studies, particularly in developing better fine-tuning techniques for human mobility modeling.