UrbanKGent: A Unified Large Language Model Agent Framework for Urban Knowledge Graph Construction

Authors: yansong  ning
hao  liu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 14:20:46

Abstract: Urban knowledge graph has recently worked as an emerging building block to distill critical knowledge from multi-sourced urban data for diverse urban application scenarios. Despite its promising benefits, urban knowledge graph construction (UrbanKGC) still heavily relies on manual effort, hindering its potential advancement. This paper presents UrbanKGent, a unified large language model agent framework, for urban knowledge graph construction. Specifically, we first construct the knowledgeable instruction set for UrbanKGC tasks (such as relational triplet extraction and knowledge graph completion) via heterogeneity-aware and geospatial-infused instruction generation. Moreover, we propose a tool-augmented iterative trajectory refinement module to enhance and refine the trajectories distilled from GPT-4. Through hybrid instruction fine-tuning with augmented trajectories on Llama 2 and Llama 3 family, we obtain UrbanKGC agent family, consisting of UrbanKGent-7/8/13B version. We perform a comprehensive evaluation on two real-world datasets using both human and GPT-4 self-evaluation. The experimental results demonstrate that UrbanKGent family can not only significantly outperform 31 baselines in UrbanKGC tasks, but also surpass the state-of-the-art LLM, GPT-4, by more than 10% with approximately 20 times lower cost. Compared with the existing benchmark, the UrbanKGent family could help construct an UrbanKG with hundreds of times richer relationships using only one-fifth of the data. Our data and code are available at https://github.com/usail-hkust/UrbanKGent.

Main paper: 
 introduction: 
 Urban Knowledge Graph (UrbanKG) aims to model intricate relationships and semantics within a city by extracting and organizing urban entities (e.g., POIs, road networks, etc.) into a multirelational heterogeneous graph [1] . As an emerging building block, multi-sourced urban data are widely used to construct an UrbanKG to provide critical knowledge for various knowledge-enhanced urban downstream tasks, such as traffic management, pollution monitoring, and emergency response [2, 3, 4, 5] . UrbanKG has gradually become an essential tool of the modern smart city.In prior literature, many efforts have been devoted to urban knowledge graph construction (Ur-banKGC) using massive urban data sources. In particular, one commonly used approach [6, 7, 8] is to extract entities from structured urban data (e.g., geographic data, city sensor data, and traffic data) and define the relationships between obtained urban entities based on manually designed rules or patterns. However, these approaches suffer heavy reliance on a deep understanding of the application domain and are labor-intensive. Recently, inspired by the success of the Large Language Models (LLMs) in knowledge graph construction [9, 10, 11] , the LLMs have been adopted to improve Ur-banKGC. For instance, GeoLM [12] is pretrained on crowdsourced geographical corpus for geospatial entity recognition and relation extraction. K2 [13] retrains Llama-2-7B model on manually processed and filtered geoscience text corpus for geospatial relation extraction. Nevertheless, these works rely on high quality corpus annotation and computational extensive model retraining, which may discourage researchers from adopting UrbanKG for their own work.LLM agent [14, 15] has recently emerged and shown remarkable zero-shot capability for autonomous domain-specific task completion. For example, Voyager [16] is a LLM-powered agent for zero-shot game exploration without re-training, and LLMLight [17] is a traffic signal control agent with zero-shot LLM reasoning ability. These studies motivate us to construct tailored LLM agents to address the aforementioned limitations in UrbanKG construction.In fact, constructing an LLM agent compatible with various UrbanKGC tasks is a non-trivial problem due to the following two challenges: (1) Challenges 1: How to adapt LLMs for UrbanKGC? LLMs may not align well with the specific task due to the gap [18] between the natural language processing corpus for training LLMs and the domain-specific corpus in urban domain [19] . For example, the urban text data is usually heterogeneous and contains multifaceted urban knowledge (e.g., spatial, temporal, and functional aspects) [13] . As shown in Figure 1 (a), the text description of "Columbia University" reflects its geographic spatial locations (i.e., spatial relationship), construction timelines (i.e., temporal relationship), and how it provides educational service for the city (i.e., functional relationship). LLMs may require a tailored alignment to understand heterogeneous urban relationships to extract these urban spatial, temporal, and functional relations accurately.(2) Challenges 2: How to improve the capacity of LLMs for UrbanKGC? The effectiveness of LLMs for urban knowledge graph construction is restricted by their feeble numerical computation capacity [20, 21] , leading to their disability in complex geospatial relationship extraction [22, 23] . However, the urban geospatial relationship plays a vital role in urban semantic modeling [12] and has been widely incorporated in previous UrbanKGs [8, 24] . As can be seen in Figure 1 , extracting "disconnected" relation between the geo-entity "Columbia University" and "Empire State Building" is useful for urban geo-semantic modeling. Accurately extracting such geospatial relationship demands necessary geospatial computing (e.g., utilizing latitude and longitude for distance calculation) and reasoning (i.e., deriving calculation results for geospatial relation reasoning) capabilities. It is appealing to improve the geospatial computing and reasoning ability of LLMs to satisfy the UrbanKGC task requirement.To address the aforementioned challenges, in this study, we propose UrbanKGent, a unified LLM agent framework for automatic UrbanKG construction. Figure 2 illustrates the overview of Ur-banKGent. For a given city, we first generate a knowledgeable instruction set for UrbanKGC tasks (relational triplet extraction and knowledge graph completion) from urban geographic and text data sources. By heterogeneity-aware and geospatial-infused instruction generation, as shown in Figure 1 (a), various urban spatiotemporal relationship knowledge can be encoded into the instruction, which facilitates alignment between LLMs with the target UrbanKGC tasks. Moreover, we propose a tool-augmented iterative trajectory refinement module to enhance and refine the trajectory derived by distilling GPT-4 with the above constructed instructions. Based on geospatial tool augmentation and self-refinement, the deficiency of LLMs in geospatial computing and reasoning could be alleviated, and unfaithful trajectories could be filtered out. Finally, we perform hybrid instruction fine-tuning based on the enhanced and refined trajectories on Llama 2-7/13B and Llama 3-8B variants [25] by using LoRA [26] . The obtained UrbanKGent agent including 7/8/13B version, is feasible for completing multiple UrbanKGC tasks cost-effectively without extra GPT-API cost. We conduct comprehensive experiments on two UrbanKGC tasks in two metropolises (New York City and Chicago) using both human evaluation and GPT-4-based self-evaluation. The empirical results validate the effectiveness of the proposed LLM agent for completing various UrbanKGC tasks. Moreover, the obtained Ur-banKGent family (7/8/13B version) could help extract the same scale of triplets and entities of existing UrbanKG benchmark [8] using only one-fifth of data, and even expand the types of relations by hundreds of times.Our contributions are summarized as follows: (1) We propose the first UrbanKGC agent framework UrbanKGent and UrbanKGent family to provide real-world UrbanKGC service, offering new opportunities to advance UrbanKG studies. (2) We propose a knowledgeable instruction generation module and a tool-augmented iterative trajectory refinement method, which align LLMs to UrbanKGC tasks and compensate for their geospatial computing and reasoning inability. ( 3) Extensive experiments on two real-world datasets validate the effectiveness of proposed framework and uncover its exceptional performance across UrbanKGC tasks.2 UrbanKGC Data DescriptionQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: In the end of abstract and introduction, the main claims are made.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
data collection: 
 We first acquire urban knowledge for two large cities New York City and Chicago from two data sources. Table 1 summarizes the statistics of the raw datasets.The geographic data provides critical urban spatial structure information and functional semantics, which has been widely used in previous UrbanKG studies [8, 24, 27, 28] .Area-Of-Interst (AOI) Data. AOI data describes the urban spatial area structure, including urban commercial areas (e.g., shopping centers), residential areas (e.g., communities), and so on. In this work, we first follow UUKG [8] to acquire the AOI name and geometry value from NYC Gov3 and CHI Gov 4 . Next, we use the AOI name to search their text description from Wikipedia and C45 dataset. Each AOI record contains an AOI name, a polygon geometry value, and a text description. For example, ["Jamaica Bay", polygon (-73.86 40.58, ...), "Jamaica Bay is an estuary ..."] is the record of the AOI "Jamaica Bay" with geometry value and text description.Road Network Data. Road data describes the urban spatial network, including urban motorways, overpasses, and so on. We first follow [8] to obtain the road name and geometry value from Open Street Map (OSM 6 . Then, following the same text acquisition operation in AOI data, we crawl the textual description of each road record from Wikipedia. Each road record contains a road name, a linestring geometry value, a road type and a text description. For example, ["Central Park Avenue", linestring (-73.87 40.90, ...), primary, "Central Park Avenue is a boulevard in ..."] describes the primary road named "Central Park Avenue" with a linestring geometry value and its textual description.Point-Of-Interest (POI) Data. POI data represents different urban functions (e.g., residential and commercial), which have been widely adopted in many recent UrbanKG works [24, 6, 8] . We first follow [8] to obtain the POI name, and geometry value from OSM. Then the textual description of each POI record could be crawled from Wikipedia following the similar process. Each POI record contains a POI name, a coordinate geometry, a POI type, and a text description. For example, The text data provides rich contextual knowledge of the city space from different perspectives (e.g., the spatial context) [13] , and it plays an important role in geospatial understanding. In this work, we collect two types of text corpus.Review Data. The review of urban places provides commercial information that citizens use to make business decisions [29] , playing a critical role in urban knowledge distillation. We collect review data from Google Map 7 . Specifically, we first manually split the city into multiple rectangular regions, then we utilize the Google Map API to query the places contained within each region and their reviews. Each review record contains a place name, a coordinate geometry value, a rating, and a text review. For example, ["Lifestyles Academy Inc", coordinate (-87.87 41.65), 4.9, "Very nice organization and ..."] is the review record of place "Lifestyles Academy Inc".Web Page Data. The web page data works as the general text corpus for the city, and it contains rich geoscience knowledge that has been utilized in recent urban entity and relation extraction studies [13] . We collect web page data from the Google search engine. Specifically, we first input the name of the crawled AOI, Road, and POI record into Google. Then we concatenate the textual sentences of the top 10 retrieved web pages. Each web page record contains a long urban text description.Before constructing the UrbanKGC dataset, we first preprocess the raw datasets. We filter out AOIs, roads, POIs, reviews, and web pages whose crawled textual descriptions are null value, too short (e.g., less than ten word description) or meaningless (e.g., just repeating the POI name). In addition, we remove irrelevant information from the text description, such as non-English characters, non-ASCII gibberish, website addresses, and so on. More details can be found in Appendix A.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The limitations are discussed in Appendix F Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminary: 
 This section presents the UrbanKGC task definition and provides task analysis.Before diving into the technical details, we first introduce the definition of UrbanKG:Definition 1 UrbanKG. The UrbanKG is defined as a multi-relational graph G = (E, R, F), where E, R and F is the set of urban entities, relations and facts, respectively. In particular, facts are defined as F = {⟨h, r, t⟩ | h, t ∈ E, r ∈ R}, where each triplet ⟨h, r, t⟩ describes head entity h is connected with tail entity t via relation r.The UrbanKG encodes diverse urban semantic knowledge by connecting urban entities into a multirelational graph. This work aims to construct an UrbanKG from collected unstructured text data. We decompose the UrbanKG construction (UrbanKGC) process into two sequential knowledge graph construction tasks, namely relational triple extraction [10] and knowledge graph completion [11] . We first provide the basic definition for these two subtasks, and then introduce the problem formulation of this work.Relational Triplet Extraction (RTE). Given the unstructured texts, this task achieves joint extraction of entities and their relations [10] which are in the form of a triplet ⟨h, r, t⟩. For instance, given the urban text sentence "Columbia University is a private Ivy league research university in New York City.", this task aims to identify two entities "Columbia University" and "New York City" and their relation "locate-in", described as triplet <Columbia University, locate-in, New York City>.Knowledge Graph Completion (KGC). Given a head entity h and a tail entity t, this task is to predict the missing relation between them [11] . For instance, given the head entity "Columbia University" and the tail entity "Empire State Building", this task is to predict that their missing relation, e.g., "disconnected", described as triplet <Columbia University, disconnected, Empire State Building>.Given the urban unstructured text data, the desired output is an UrbanKG G. In this paper, this problem is decomposed into two sequential subtasks: (1) Relational Triplet Extraction: the first task extracts relational triplet ⟨h, r, t⟩ from the urban text data. The output of RTE task isG 1 =(E, R 1 , F 1 ), where E and R 1 is the set of extracted entities and relations, while F 1 is the set of extracted triplets.(2) Knowledge Graph Completion: for the given head entity h and tail entity t in G 1 , the second task is to predict the geospatial relationship8 between them. The output of this task isG 2 =(E, R 2 , F 2 ), where R 2 and F 2 is the set of completed relations and triplets. By sequentially completing the above two tasks, we can obtain the constructed UrbanKGG = (E, R 1 ∪ R 2 , F 1 ∪ F 2 ).As shown in Figure 1 , we qualitatively find that LLMs lack urban heterogeneous relationship understanding ability and experience in geospatial computing and reasoning difficulty when adopting it for UrbanKGC tasks. This subsection presents a quantitative analysis of these two challenges.Heterogenous Relationship Understanding. The ability to understand heterogeneous relationships is ubiquitous in distilling knowledge from the massive urban corpus. For example, the text description in Figure 1 illustrates a place from spatial location, temporal time, and functional aspects. Capturing these heterogeneous semantics is important for urban knowledge distillation. We perform quantitative analysis by randomly sampling 50 urban text data and then prompt GPT-4 to complete relational triplet extraction by providing only the basic task description. As shown in Figure 3 (a), we find the LLMs experience serious misjudgment (i.e., extract wrong triplets or miss the triplet) on urban spatial, temporal, and functional triplet extraction. This indicates the limited capacity of LLMs to understand heterogeneous relationships.Geospatial Computing and Reasoning. Geospatial computing and reasoning techniques are widely used in many previous UrbanKG studies [8, 24] for urban geospatial relation extraction. In recent works [23, 31] , the geospatial skills of LLMs have also been demonstrated to lack geospatial awareness and reasoning ability [22] . To identify potential limitations, we quantitatively investigate how LLMs can perform on geospatial relation completion tasks. Specifically, we construct 100 head and tail entity pairs, covering five geospatial relations in the KGC task, and then prompt GPT-4 to predict with basic task description and geospatial relation candidates. As shown in Figure 3 (b), we find that GPT-4 performs poorly on five geospatial relation completion. This further validates the disability of LLMs in geospatial computing and reasoning.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: This paper focus on domain application, no theory assumptions or proofs provided in this paper. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
urbankgc agent construction: 
 This section presents the proposed UrbanKGC agent construction framework. We first construct the knowledgeable instruction to adopt LLMs for two UrbanKGC tasks, including relational triplet extraction (RTE) and knowledge graph completion (KGC). Figure 4 (a) illustrates the overview of the instruction construction process of these two tasks.Heterogeneity-aware Instruction Generation for Relational Triplet Extraction. As discussed in Section 3, the urban text contains diverse heterogeneous relationships, thus we consider multiple views with both urban entity and relation definition for relational triplet extraction. In particular, we construct a multi-view instruction template for the urban relational triplet extraction, including spatial view, temporal view, and functional view. Each view is a multi-turn question-answer dialog [32] consisting of entity recognition, relation extraction, and triplet extraction module.For the spatial view, we devise a two-turn dialog to align LLMs for spatial triplet extraction. In the first turn, we inject spatial entity and relation definition into the instruction template to guide LLMs to understand spatial characteristics and then extract potential spatial entities (e.g., University) and relations types (e.g., locate-in). In the second turn, the extracted types are explicitly fed into the instruction template for spatial triplet extraction. Intuitively, the spatial view allocates dedicated urban knowledge for LLMs to extract urban spatial relationships. Similarly, we construct the temporal view and functional view for corresponding temporal and functional triplet extraction, independently.Geospatial-infused Instruction Generation for Knowledge Graph Completion. Despite heterogeneity-aware instruction enabling LLMs to extract urban triplets from various perspectives, the geospatial relationship between geospatial entities cannot be directly extracted. Therefore, we introduce a geospatial-infused instruction generation module to guide LLMs to complete missing geospatial relationships.First, we incorporate the geometry information (i.e., the latitude and longitude) of geo-entities into instruction, so that the LLMs can utilize these geospatial values for relation inference. Second, we add the geospatial relationship definition to the instruction to guide LLMs in understanding the geospatial relationship definition. Intuitively, LLMs can refer to geospatial knowledge and make practical solutions for the KGC task. We provide the detailed instruction template in the Appendix B.With the initial UrbanKGC instructions constructed, the following step is to generate reasoning trajectories [33] , which will be used to fine-tune LLMs tailored to UrbanKGC task. Specifically, we follow FireAct [34] and use Chain-of-Thought (CoT) [35] , a gradient-free technique, to prompt GPT-4 (i.e., add prompt trigger "Let's think step by step" at the end of RTE and KGC instructions template) to generate the reasoning trajectories for UrbanKGC tasks.The generated CoT trajectories could provide a step-by-step reasoning solution for UrbanKGC tasks. Nevertheless, the complex geospatial relationships cannot be easily extracted as discussed in Section 3 and recent geospatial reasoning works [31, 23, 22] . Therefore, we introduce a tool invocation module to guide LLMs to invoke tailored external geospatial tools [36] to enhance their geospatial computing and reasoning capacity for UrbanKGC tasks.We conduct two sequential procedures: tool invocation for geospatial computing support and trajectory deliberation for reasoning enhancement.Tool Invocation. First, we construct a geospatial reasoning toolkit (e.g., distance calculation, eight interfaces in total shown in Table 6 ) by prompting GPT-4 for self-programming. Then, we construct tailored prompts to guide LLMs to invoke these interfaces. Specifically, the prompt is concatenated with an illustrative description of the function of each geospatial tool and a task instruction (i.e., "Which types of tool interface you need"). Intuitively, the external tool allocates calculation results for LLMs to infer missing geospatial relation. The toolkit description can be found in Appendix C.Trajectory Deliberation. After manipulation with external tools, we prompt LLMs to refine uncertain reasoning steps based on these obtained manipulation results. Specifically, we construct the prompt by concatenating with manipulation results (e.g., the distance and geohash value of geo-entity) and a task instruction (i.e., "Please refine your reasoning process"). After feeding the prompt into GPT-4, the enhanced trajectory is obtained. Detailed prompt information can be found in Appendix C.Despite tool-augmented deliberation improving the geospatial computing and reasoning ability of LLMs, enhanced trajectories may not all be faithful [37] . To alleviate potential error and ensure the trajectory quality [38] , we refine these trajectories via an iterative self-refinement mechanism [39] . Specifically, we iterate two sequential blocks: (i) Trajectory verifier: given the trajectory, the verifier aims to provide feedback for refining the reasoning process; (ii) Trajectory updater: given the trajectory and feedback, the updater will further refine the current trajectory based on the feedback.Trajectory Verifier. We construct a tailored prompt to ask LLMs to generate feedback. Specifically, we use a simple but effective trigger ("Judge whether all extracted triplets are correct and provide improvement suggestion") to prompt LLMs to provide feedback. If the trajectory no longer requires modification, we let LLMs respond with "This is a faithful trajectory". Such a verification step lets LLMs make reflections and improve the correctness of the trajectory.Trajectory Updater. The updater utilizes provided feedback to refine the current trajectory via prompt trigger "Follow suggestion to refine the reasoning process". Intuitively, the feedback may address multiple aspects (e.g., missed triplet in the RTE task or unfaithful reasoning process in the KGC task) of the unfaithful trajectories.We iterate the trajectory verifier and updater until the predefined stopping condition is satisfied. The stopping condition is determined by either meeting the maximum number of iterations (we set it at three to avoid excessive cost) or when the verifier confirms all trajectories are faithful. Upon meeting the stopping condition, we use the last refined trajectory for further fine-tuning. Detailed prompt information can be found in Appendix C.To construct a cost-effective UrbanKGC agent, we further utilize trajectories (generated by GPT-4) to fine-tune a smaller open-source LLM for faster inference speed and lower cost (i.e., prompting GPT-4 for UrbanKGC is expensive). Specifically, we finetune the LLM via the mixed-task instruction-tuning strategy [33] . The goal is to enhance the LLMs' capabilities in diverse UrbanKGC tasks.Mixture Training. Set the base language model as M, and P M (y | x) represents the probability distribution of response y given instruction x. We consider the trajectory set on two UrbanKGC tasks, i.e., D RT E and D KGC . Since both the instruction and the target output are formatted in natural language, we can unify the training into an end-to-end sequence-to-sequence way. Formally, the optimization process aims to minimize the loss of language model M as follows:EQUATIONwhere x and y represent the instruction input and instruction output in the trajectory, respectively.Training Setup. We choose the chat version of open-sourced Llama 2-7/13B and Llama-3-8B as our backbone models, and fine-tune Llama using LoRA strategy [26] .Via hybrid instruction fine-tuning, the obtained LLM UrbanKGent can be trained to follow the instructions to finish the UrbanKGC task. We prompt UrbanKGent to complete UrbanKGC tasks by following the pipeline shown in Figure 4 . For the RTE task, we sequentially execute entity recognition, relation extraction, and relational triplet instruction generation, iterative self-refinement and output the extracted triplets. For the KGC task, we sequentially execute KGC instruction generation, external tool augmentation, iterative self-refinement block, and finally output the completed triplets.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: Researcher could follow the provided code to reproduce experimental results. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. • The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experimental settings: 
 Dataset. In this work, two sequential tasks (i.e., RTE and KGC) of UrbanKGC are within an openworld setting (i.e., no predefined ontology) [40, 41] . We construct the RTE and KGC datasets of NYC and CHI by sampling uniformly from five raw data in Table 1 , respectively. As shown in Table 2 , we first construct two small datasets (i.e., NYC-Instruct and CHI-Instruct) for instruction fine-tuning and two middle datasets (i.e., NYC and CHI) to validate the performance of the constructed UrbanKGC agent. The remaining data works as the large-scale UrbanKGC dataset (i.e., NYC-Large and CHI-Large) in real-world scenarios shown in Table 5 . The three types of datasets are non-overlapping to prevent data leakage. More dataset construction details are in Appendix A.Baseline Methods. We provide a comprehensive comparison of our method with existing paradigms:(1) End-to-end Models: For the zero-shot RTE task, we utilize the end-to-end generation model RelationPrompt [42] and PRGC [10] . For the KGC task, we fine-tune KG-BERT [43] and KG-T5 [44] with the QA pairs constructed from the self-instruct dataset. Implementation and Detail Settings. In our experiment, we select Vicuna [46] , Alpaca [47] , Mistrial [48] , Llama-2 [49] , Llama-3 [50] , GPT-3.5 [51] and GPT-4 [51] as the backbone LLM M.All experiments are conducted on eight NVIDIA A800 GPUs. For the GPT-3.5 and GPT-4, we adopt the gpt-3.5-turbo-16k-0613 API and gpt-4-0613 API.Evaluation Protocol. Since UrbanKGC tasks in this work follow an open-world setting where labels are not visible, the classical metric (e.g., F1 and Hits@10) is not applicable. In this work, we regard evaluation as the binary classification, i.e., if the extracted triplet in RTE task is correct and if the completed relation in KGC task is correct. We follow recent LLMs-based KGC works [11] to employ accuracy as an evaluation metric. To make a comprehensive evaluation of the experimental results, we employ both of the human evaluation and GPT evaluation, which has been widely used in many LLM studies [52, 45] . For Human Evaluation, we employ human annotators to evaluate the results on 200 random samples. As for the GPT Evaluation, we use GPT-4 to evaluate the model performance on the full data to escape intensive labor. In this work, the GPT-4's evaluation has been demonstrated to be consistent with the human evaluation. Detail is in Appendix D.The performance results are reported in Table 3 . As can be seen, the constructed agent outperforms all thirty-one baseline models on two UrbanKGC datasets. Specifically, the UrbanKGent-13B achieves (15.56%, 14.29%, 14.89%, and 11.90%) improvements compared with the state-of-the-art GPT-4 with the same inference pipeline on NYC. The improvements on CHI are (15.22%, 17.07%, 13.46%, and 13.95%), respectively. Moreover, the UrbanKGent-7/8B also achieve comparable performance compared with the GPT-4.Meanwhile, we observe that the zero-shot LLMs perform poorly in the UrbanKGC tasks, even using GPT-4. In addition, although the demonstrations provided by In-context-learning can incorporate the UrbanKGC task information, the performance gain is limited. Besides, we find that fine-tuning LLMs can make obvious improvements in the overall performance. Through vanilla fine-tuning, the Llama-2-7/13B and Llama-3-8B could achieve comparable performance with GPT-3.5 under the ZSL settings.Moreover, although the various LLM backbones using the UrbanKGent inference pipeline perform slightly worse than the vanilla fine-tuning method, they could obtain better performance compared with zero-shot reasoning and In-context learning paradigms. Such results demonstrate the benefit of knowledgeable instruction design and external tool innovation, but also indicate its performance bottleneck. As a deeper exploration, our work fills this gap through hybrid instruction fine-tuning, and the fine-tuned UrbanKGC agents, whether 7B, 8B or 13B, can achieve state-of-the-art performance in UrbanKGC tasks.We provide an in-depth analysis of the proposed UrbanKGent framework in Appdendix E.2.We first derive UrbanKGent-13B for initial UrbanKGs acquisition in New York City and Chicago.After proper filtering and merging of the triplets, we obtain two large-scale UrbanKGs shown in Table 4 . Compared with existing UrbanKG benchmark [8] , we only use roughly one-fifth of the data for constructing the UrbanKGs with the same scale of triplets and entities, and even expanding the variety of relationships to a hundred times the original types. Moreover, we also provide efficiency analysis in Figure 5 . As can be seen, UrbanKGent-13B achieves lower inference speed in latency and reduce the cost by roughly 20 times in both of RTE and KGC tasks. More details is in Appendix E.3. 
related work: 
 Domain-Oriented Agent Construction. The concept of language agent [34] has become very popular recently, and a variety of LLM agents targeting different domains have been proposed. For example, Voyager [16] is constructed for automated game exploration, WebGPT [17] is an HTML agent for diverse document understanding tasks, LLMLight [53] constructs a language agent for transportation domain, K2 [13] , GeoGalactica [19] and GeoLLM [12] propose to re-train language agent for geospatial semantic understanding. In addition, many recent works like Auto-GPT [54] and CAMEL [55] aim at proposing an autonomous agent framework for agent construction. Nevertheless, there is still no UrbanKGC agent construction framework for the urban computing domain.LLMs for Knowledge Graph Construction. Recently, the advent of LLMs [56] invigorated the field of NLP. Many studies have begun to explore the potential of LLMs in the domain of KG construction. For example, [32, 57] finds that transforming the NER and RE task into a multi-turn question-answering dialog could improve the model performance. [9] explicitly derive syntactic knowledge to guide LLMs to think, which could develop the performance of NER. Despite these LLM-driven KG construction methods [58, 40] in general domains being widely investigated, KG construction in urban domain still remains an open challenge [59] .Urban Knowledge Graph. Urban knowledge graph has been proven useful in various urban tasks, such as traffic flow prediction [60, 61, 27, 62] , mobility prediction [6] , site selection [7] , city profiling [63] , crime prediction and so on [8, 64, 65] . Their common approach involves manually extracting urban entities and defining urban relations to construct an urban knowledge graph. For example, [6] construct a dedicated spatiotemporal knowledge graph regarding trajectory and timestamp as entities to improve trajectory prediction and [7] construct user check-in relations to help mobility prediction. Nevertheless, existing UrbanKGs heavily rely on manual design, leading to high labor costs. 
conclusion: 
 In this work, we proposed UrbanKGent, the first automatic UrbanKG construction agent framework with LLMs. We first constructed a knowledgeable instruction set to adopt LLMs for different UrbanKGC tasks. Then, we proposed a tool-augmented iterative trajectory refinement module to facilitate the instruction tuning of various large language models. Extensive experimental results demonstrate the advancement of UrbanKGent in improving UrbanKGC tasks. The horizontal and vertical coordinates are longitude and latitude, respectively. The blue and red polygons stand for entities with the polygon geometry, the purple line string stands for the entities with linestring geometry and the green point is for the coordinate entities. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: In the experimental setting section 5.1, detailed computing resources explanation is provided.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: This paper follows the NeurIPS Code Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: In the end of introduction and agent application section 5.3, we discuss how can our proposed LLM agent to be deployed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The data and models used in this paper are opensourced, there is no critical issue in model and data.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The researcher can follow CC-BY 4.0 license to use the data and code in this paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: X Y X Y X Y X Y X Y X Y Y X Y X X DC Y X EC Y X PO Y X EQ Y X IN YFigure 6 : Given two geo-entities X and Y, the illustrative visualization of five types of RCC relationships. In this work, we consider entities with coordinate geometry to a circles with very small radii.This section presents detailed UrbanKGC dataset statistic information for relational triplet extraction (RTE) and knowledge graph completion (KGC) tasks. Since two sequential tasks (i.e., RTE and KGC) of UrbanKGC are within an open-world setting (i.e., no predefined ontology) [40, 41] . Therefore, for the RTE task, every data record is an urban text without the triplet label. For the KGC task, every data record is a quadruple (i.e., head entity name, head entity geometry, tail entity name, tail entity geometry) without the geospatial relation label.RTE Dataset. To facilitate the understanding of the constructed RTE dataset, we summarize the distribution of urban textual corpus in the six RTE datasets. As shown in Table 5 , the entity distribution and textual statistics of the instruct dataset and test data are similar. In addition, the record in dataset (i.e., NYC-Instruct and CHI-Instruct) used for instruction tuning is not overlapping with that in test dataset (i.e., NYC and CHI) and the real-world application dataset (i.e., NYC-Large and CHI-Large). This can avoid potential data leakage issues.KGC Dataset. As for the KGC dataset, we provide illustrative visualization of the five RCC relationships [30] in Figure 6 for better understanding. Specifically, the disconnected (DC), externally connected (EC), partially overlapping (PO), equal (EQ), tangential and non-tangential proper parts (IN) together depict the basic geospatial relationship between urban entities. Moreover, to facilitate the understanding of the constructed KGC dataset, we visualize the geometry range of the head entity and the tail entity in four small KGC datasets. Due to the large amount of data and the display overlapping between entities and entities during visualization, we will not show the visualization results of NYC-Large and CHI-Large dataset, but the pattern is similar. As can be seen in Figure 7 , the entity distribution in the instruct dataset (i.e., NYC-Instruct and CHI-Instruct) and test dataset (i.e., NYC and CHI) are similar. The pattern in NYC-Large and CHI-Large is similar. Due to the KGC task follows a zero-shot setting, we are unable to provide accurate distribution proportions of the 5 RCC relationships in the four datasets. However, it is intuitive can be seen that all four datasets contain the five RCC relationships shown in Figure 6 , which provides a guarantee for the practical significance of KGC task in this work.This section presents the detailed instruction template of the proposed UrbanKGent framework. Specifically, Figure 8(a-b ) provides the detailed instruction template of knowledgeable instruction generation module and tool-based trajectory augmentation. The iterative trajectory self-refinement module is achieved by trajectory verifier and updater in Figure 8 (c).We provide the detailed instruction template of all the baseline models in this work.LLMs-based Zero-shot Reasoning Methods. We only provide task descriptions for the zero-shot reasoning method. The detailed instruction template is shown in Figure 9 .LLMs-based In-context Learning Methods. We first construct several few-shot demonstrations via the chain-of-thought prompting techniques [35] , which is popular for automatic demonstration generation. The detailed instruction template is shown in Figure 9 . Then, we add these demonstrations before the test question, and the detailed instruction template is shown in Figure 9 .Vanilla Fine-tuning Methods. We only provide task descriptions for vanilla fine-tuning methods without demonstrations. The detailed instruction template is shown in Figure 9 .UrbanKGent Inference Methods. These baseline models follow the same inference pipeline shown in Figure 8 .This work constructs a geospatial reasoning toolkit (eight interfaces in total in Table 6 ) by asking GPT-4 for self-programming. We obtained geospatial toolkit for point, linestring, and polygon geometry objects, supporting distance calculation and Geohash encoding for two geographical objects, and can also determine basic spatial relationships between two geometric objects, such as containment and intersection. For each interface, we provide its name and function description (e.g., geohash encoding in Table 6 and Figure 8b ) into the instruction template. This section presents the detailed evaluation process and examples to help the reader have a better understanding: (1) Human Evaluation. We employ human annotators to evaluate the results on 200 random samples. For the relational triplet extraction task, we first manually annotate the triplet label for each sample. Then, we manually evaluate the correctness of each triplet [40] based on annotation and calculate the accuracy value. For the knowledge graph completion task, we follow [11] to manually label the response as correct or wrong, and calculate the accuracy. (2) GPT Evaluation.Recently, many studies [52, 45] adopt LLM-based evaluation for open-domain tasks and empirically demonstrate that GPT-4's evaluation and human evaluation can be consistent [66] . In this work, we also use GPT-4 to evaluate the model performance on the full data to escape intensive labor. Specifically, given an UrbanKGC instruction and results, we prompt GPT-4 to return the confidence score and the justification (i.e., True/False), which will be further used to calculate the accuracy.Moreover, we provide a comprehensive analysis to demonstrate why GPT and Human evaluations are highly aligned.For the relational triplet extraction (RTE) task, we provide an evaluation example in Table 7 . Given the urban text sentence "Columbia University is a private Ivy league research university in New York City.", the human annotators are required to first label the triplet contained in this urban text, described as triplet <Columbia University, locate-in, New York City>. Based on the label, then, they are instructed to evaluate how many true triplets and false triplets in the results from the models. Finally, they will fill out the evaluation form (i.e., the number of the true triplets and the number of the false triplets). We will calculate the accuracy of results based on these annotated forms.For the knowledge graph completion (KGC) task, given the head entity <Columbia University, Point (40.8075, 73.9626)> and the tail entity <Empire State Building, Point (40.7484, 73.9857)>, the human annotators are first required to complete their missing geospatial relationship from the five relation candidate (i.e., DC, EC, PO, EQ and IN). Specifically, the annotation could be achieved by manually visualizing the location of two entities given on the map and following the RCC relation rule in Figure 6 to determine their geographical relationships. Finally, they will fill out the evaluation results (i.e., True/False). We will calculate the accuracy based on these evaluation results.For the relational triplet extraction (RTE) task, we provide an evaluation example in Table 7 . Given the urban text sentence "Columbia University is a private Ivy league research university in New York City.", and the model extraction results (i.e., <Columbia University, locate-in, New York City>), we directly prompt GPT-4 to fill out the evaluation form (i.e., the number of the true triplets and the number of the false triplets). Then, the accuracy could be obtained based on these self-evaluated results.For the knowledge graph completion (KGC) task, given the head entity <Columbia University, Point (40.8075, 73.9626)> and the tail entity <Empire State Building, Point (40.7484, 73.9857)> and completed geospatial relationship, we directly prompt GPT-4 the justify if the results are correct. Specifically, we will first explicitly call eight external geospatial tools in Table 6 , and combine all these eight calculation results into the prompt. Then we feed these prompts into GPT-4 to help it make a comparable accurate evaluation. Finally, we obtain the evaluation results (i.e., True/False) for every record. We will calculate the accuracy based on these evaluation results.However, such an evaluation process carries risks as we cannot guarantee the infallibility of GPT-4. To ensure the validity of the evaluation, we have conducted extensive experiments to demonstrate that the GPT evaluation method aligns well with human evaluation.We conduct experiments to assess the consistency between GPT-4's evaluation results and human evaluation results. In this work, we follow GPTScore [67] to use Spearman coefficient [68] to investigate the correlation between GPT evaluation and human evaluation. Based on the evaluation results in Tabel 3, we can conduct correlation analysis separately by the types of model and paradigm.Comparison of LLM Backbones. As shown in Figure 10 , we observe that the Spearman correlation coefficients on all LLM backbones are greater than 0.8, which means the human evaluation and GPT evaluation on these five LLM backbones is highly correlated.Comparison of LLM Paradigm. Moreover, we also conduct correlation under different LLM paradigms for UrbanKGC tasks. As can be seen in Figure 11 , all four types of paradigm have a higher Spearman value above 0.85. Therefore, deriving the GPT evaluation method is also applicable to different LLM paradigms when finishing UrbanKGC tasks as its evaluation aligns well with that of the human being.Analysis of Evaluation Repeat. Nevertheless, the above correlation analysis is based on results in Table 3 , which are obtained after only single evaluation. To deeply understand the evaluation mechanism, We repeatedly instruct GPT-4 to generate 1, 3, 5, and 10 evaluation results to understand the variance of the evaluation method. To save the cost, we perform the above analysis using UrbanKGent-7B only on the NYC dataset. The results are shown in Table 8 , although the coefficient value will decrease as the number of repeated experiments increases. However its value has always remained within a high correlation range (roughly 0.85).In this section, we aim to perform more comprehensive analysis of our proposed UrbanKGent agent and hope to answer the following research questions. (1) RQ1: How does the constructed Table 8 : The average Spearman correlation value between human evaluations and GPT-4 evaluations. "Repeat X times" refers to instructing GPT-4 to generate judgments X times, and adopting the answer that appears most frequently (e.g., True/False for the KGC task and Number of the true triplet for the RTE task) as the final decision. UrbanKGent perform compared with existing paradigms on larger real-world dataset? (2) RQ2: How do different components (e.g., the knowledgeable instruction generation) affect the performance?(3) RQ3: How the complexity and efficiency of proposed UrbanKGent framework? (4) RQ4: How about the UrbanKGent inference trajectories when completing UrbanKGC tasks? (5) RQ5: How can the constructed UrbanKGent provide application service to real-world scenarios?As shown in Table 9 , we derive UrbanKGent-13B for urban knowledge graph construction using constructed large-scale dataset in NYC and CHI. Specifically, we directly use the text record in NYC-Large and CHI-Large for the relational triplet extraction task. Then, we randomly sample the head-tail entity pairs (both of head and tail entities contain geometry information) from these triplets for knowledge graph completion. Nevertheless, iterating all head-tail entity pairs is a time-consuming task, so we just construct a KGC dataset consistent with the sacle of the RTE dataset. By performing RTE and KGC tasks, we obtain two large-scale UrbanKGs shown in Table 4 . Compared with existing dataset and benchmark UUKG [8] , we can clearly observe that our agent can only use one-fifth data to construct the UrbanKG with the same scale entities and triplets, but extend the relationship types to a thousand times. It is worth noting that all construction process is completed by a LLM agent without any mannul effort. We think it is the core of this work.Moreover, we also report the performance of the RTE and KGC in NYC-Large dataset. To save the cost of GPT self-evaluation, we only choose the best approaches in ZSL, ICL, VFT and UrbanKGent Inference. The experimental results are shown in Table 9 . As can be seen, the fine-tuned UrbanKGC agent, whether 7B or 13B version, could achieve state-of-art performance on UrbanKGC tasks.We conduct an in-depth analysis of the proposed instruction generation and tool-augmented iterative trajectory refinement module on the NYC dataset. Specifically, for the RTE and KGC task, we validate the effectiveness of each block by comparing the following variants: ( ZSL × × × × ICL √ × × × VFT √ √ × × UrbanKGent Inference √ × √ √ UrbanKGent √ √ √ √UrbanKGent-7B † removes iterative trajectory self-refinement. We summarize the results in Table 10 , and obtain the following observations.First, knowledgeable instruction generation contributes to the overall performance of both RTE and KGC tasks. We observe a performance degradation by removing the knowledgeable instruction template. Second, the multi-view instruction design provides the most performance gain, which matches our intuition that the UrbanKG text contains heterogeneous relationships that can be effectively extracted by multi-view prompting design. Third, the tool invocation is very important for the KGC task, as we can observe significant performance degradation after removing the tool invocation.In addition, the iterative trajectory self-refinement brings consistent performance gain for both the RTE and KGC tasks.We make a comparison with the four paradigms to demonstrate the advantages of the constructed agent, which is shown in Table 11 . Compared with Zero-shot reasoning (ZSL), In-context Learning (ICL), Vanilla Fine-tuning (VFT), and UrbanKGent Inference, UrbanKGent can incorporate extra urban knowledge, invoke external tools and iteratively self-refine to help better complete UrbanKGC tasks.Moreover, we also provide comprehensive efficiency analysis to show the latency and cost of different models when completing UrbanKGC tasks. Specifically, we report the total inference time and cost 9of each method completing with 1,000 RTE and KGC tasks. For the cost of GPT-4 service, we first count the number of prompt token and completion token spent on 1,000 RTE and KGC tasks, and then calculate the cost based on billing standards. As for the cost of UrbanKGent-13B, we first count the GPU running time spent on 1,000 RTE and KGC tasks, and then calculate the cost based on the A800 charging standard.In addition, as reported in table 12 , we also provide detailed inference latency of UrbanKGent family when deriving them for constructing different-scale of UrbanKGs.As shown in Figure 8 , we present the detailed reasoning process of constructed UrbanKGent-13B when finishing urban relational triplet extraction and knowledge graph completion task. Since the iterative self-refinement process contains excessive text, we display the reasoning process of only one iteration.We have released the UrbanKGent family consisting of 7B, 8B and 13B version in the Huggingface. The opensourced UrbanKGent family offer urban knowledge graph construction service for the researcher in this field. We provide application example in New York and Chicago. Specifically, following the UrbanKGent Inference framework, we sequentially derive UrbanKGent-13B for relational triplet extraction and knowledge graph completion. The obtained initial UrbanKGs encodes diverse urban spatial, temporal and functional knowledge. Then, we propose to use a two-stage triplet filtering and relation merging operation to further improve the quality of constructed UrbanKGs.In the first stage, low-frequency relations (occur 5 times or less) are merged into high-frequency relations based on relation similarity threshold. The remaining low-frequency triples, whose similarity with any high-frequency relation is below the threshold will be filtered out. In the stage two, we first perform relation clustering based on the embedding of relations. Then, Within each cluster, we prompt LLM to identify semantically similar relations that can be merged into a single relation category, resulting the final set of merged relations.As shown in This work has limitation on the further application demonstration of construction UrbanKGs, although proposed UrbanKGent family could construct a UrbanKG with a hundreds relationship using only one-fifth of data. In addition, the evaluation method in this work is cost-intensive although GPT evaluation and Human evaluation has been experimentally demonstrated to be consistent. Despite the above limitations, we hope the opensource UrbanKGC agent can foster more extensive UrbanKG research and broad smart city application. In the future, we will derive extra image-modality data to further enrich UrbanKGC.NeurIPS Paper Checklisthttps://huggingface.co/usail-hkust/UrbanKGent-7B, https://huggingface.co/usail-hkust/UrbanKGent-8B and https://huggingface.co/usail-hkust/UrbanKGent-13Bhttps://www.nyc.gov/https://www.chicago.gov/https://huggingface.co/datasets/allenai/c4(https://www.openstreetmap.org/https://www.google.com/mapsWe follow GeoLM[12] to provide five RCC relationship[30] candidates: Disconnection (DC), external connection (EC), equality (EQ), partial overlap (PO), and tangential and non-tangential proper parts (IN). Details are in Appendix A.We subscribe to the NVIDIA A800 computing resources and GPT service from HKUST(GZ). Following the standard price instruction, we could calculate cost of GPT-based baselines or Llama-based baselines. 


Decision: Existing Urban Knowledge Graph Construction (UrbanKGC, which encompass relation extraction and KG completion tasks) approaches either rely on manually designed patterns/rules or annotated corpus to train LLMs. This work proposes  an agent framework UrbanKGent to solve UrbanKGC with minimal human labor and has the following components:
1) prescripted LLM process prompts:
a) triple extraction prompts based on a two-turn dialog with LLM
b) KG completion prompts infused with geospatial information
c) toolkit description/invocation prompts
d) reasoning refinement prompts
2) iterative trajectory refinement based on a verifier/critique prompt and an updater prompt
3) to achieve efficient inference smaller LLMs are fine-tuning on the larger LLMs trajectories. 

Evaluation on two real-world datasets using both human and GPT-4 self-evaluation show that UrbanKGent outperforms various LLM baselines and end-to-end models.

Reviewers recognize the importance of the application and the motivation behind the system design. This is solid work applying LLM to information extraction in the UrbanKG domain, and may set the stage for more advanced learning approaches in the future.

Reviews: review: paper_summary: paper_summary: This paper proposes a unified large-scale language model agent framework called UrbanKGent, specifically for the Construction of Urban Knowledge Graph Construction (UrbanKGC). UrbanKGent utilizes instruction generation of heterogeneous and geospatial information, as well as an iterative trajectory optimization module based on GPT-4, to further enhance the ability to extract critical knowledge from multi-source city data and effectively reduce the significant manual labor of traditional methods. By fine-tuning the hybrid instructions on the Llama 2 and Llama 3 series models, the researchers developed the UrbanKGC agent, including the UrbanKGent-7/8/13B version. Experiments on two real-world datasets show that the UrbanKGent family not only significantly outperforms 31 benchmark models on the UrbanKGC task, but is also more than 20 times more cost-effective than GPT-4, while being able to build a knowledge map of cities with hundreds of times richer relationships with less data.
strength_weakness: strength_weakness: 1. As the size of the dataset grows, the algorithmic efficiency and scalability of UrbanKGent may become an issue. Although not mentioned in the paper, in practical applications, the processing of large-scale data sets may require more computational resources and time. Although the paper notes limitations, the specifics may require more elaboration. For example, under what conditions may the model perform poorly and how these limitations affect the quality and reliability of the final knowledge graph?

2. UrbanKGent may face privacy and fairness challenges, especially when handling personal or sensitive information. I wonder what the author thinks about privacy and fairness. Although these issues are not discussed in detail in the paper, in actual deployment, if not properly addressed, they can lead to violations of personal privacy or unfairly affect specific groups.

3. The UrbanKGent framework is mainly aimed at urban knowledge graph construction tasks. I wonder if the author has considered the applicability of other types of knowledge graph construction tasks. Can the UrbanKGent framework proposed in this paper be further extended to the knowledge graph construction in other fields?
questions: questions: NA
limitations: limitations: NA
paper_summary: paper_summary: The paper presents UrbanKGent, a unified large language model agent framework for urban knowledge graph construction. The framework consists of knowledgeable instruction generation, tool-augmented iterative trajectory refinement, and hybrid instruction fine-tuning. The authors evaluate the framework on two real-world datasets using both human and GPT-4 self-evaluation. The experimental results show that UrbanKGent outperforms 31 baselines in urban knowledge graph construction tasks and achieves state-of-the-art performance.
strength_weakness: strength_weakness: 1. This paper focuses on urban knowledge graph construction tasks. However, the details of the urban knowledge graph are not clearly defined. For example, the ontology of urban KGs, entities types, and relations types are not discussed. This limits the understanding of the proposed framework for readers who are not familiar with urban knowledge graphs.

2. The KGC tasks only focus on the geospatial relations. Is there any specific reason to select these relations? Can we predict other relations using the same framework?

3. The motivation of the KGC task is not clearly explained. If the geospatial information is given, can we directly use the tool to get their relations? The paper should provide more insights into the motivation behind KGC task and the necessity of using LLMs for it.

4. The ablation study is not clearly described. The paper should provide more details on the datasets, settings, and the performance of the final model used in Table 10. Otherwise, it is hard to evaluate the effectiveness of each module.

5. The iterative trajectory self-refinement module is not well explained. Based on what criteria is the trajectory refined? How many iterations are performed? How does the performance improvement as the number of iterations increases? Moreover, if LLM lacks the knowledge about urban KGs as the motivation of this paper, how can it be a good judger? Is there any analysis about the correctness of the comments and refined trajectory?

6. The cost for fine-tuning data construction is not discussed. How much data is required for fine-tuning the model? What is the cost of collecting and labeling the data for fine-tuning?

7. In experiments, the paper should compare UrbanKGent with existing UrbanKG construction methods rather than purely focusing on the LLMs which are not tailored for this task.
questions: questions: 1. What is the ontology of urban KGs?
2. What is the motivation for selecting geospatial relations for KGC tasks and using LLMs for this? Can we predict other relations using the same framework? Can we directly use the tool to get relations if geospatial information is given?
3. Please provide more analysis of the iterative trajectory self-refinement module.
4. Settings of the ablation studies.
5. Cost for fine-tuning data construction.
6. Can we compare UrbanKGent with existing UrbanKG construction methods?
limitations: limitations: N/A
paper_summary: paper_summary: The paper introduces UrbanKGent, a comprehensive framework utilizing a large language model (LLM) for constructing urban knowledge graphs (UrbanKGs). The key components of UrbanKGent involve creating an instruction set for tasks like relational triplet extraction and knowledge graph completion, which are tailored to urban data's heterogeneity and spatial characteristics. An iterative refinement module is employed to improve and adjust trajectories derived from the GPT-4 model, enhancing the quality of the knowledge graph. The framework is then fine-tuned using these enhanced trajectories on the Llama 2 and Llama 3 models, resulting in UrbanKGC agents, available in 7/8/13B versions.
strength_weakness: strength_weakness: 1. The computational efficiency of the proposed algorithms and their scalability with the size of the dataset is not thoroughly discussed.
questions: questions: 1. As mentioned in Appendix Section D.1, human annotators are employed to fill out an evaluation form. Is it possible to provide more detailed information on human annotators, such as an exact number of human annotators?
limitations: limitations: As acknowledged in appendix section F, the paper has limitations on the further application demonstration of construction UrbanKGs and the evaluation method in this work is cost-intensive.
paper_summary: paper_summary: This paper presents a framework denoted UrbanKGent, for finetuning an LLM to assist the construction of knowledge graph construction (specifically triplet extraction relation prediction). The gist of the study is to construct an ad-hoc corpus for finetuning. In this process, a method of trajectory refinement is proposed to enhance both the effectiveness and explainability of the framework.
strength_weakness: strength_weakness: 1) It seems that the generation of the corpus, which is a key ingredient of the framework, lacks a systematic and scientific methodology for its generation.
2) The transferability is questionable – is this method useful in cities that are not finetuned?
3) The description of the roles of GPT4 and the smaller LLM that is finetuned is vague.
questions: questions: See weaknesses.
limitations: limitations: See weaknesses.

Comments: title: Response
Response to Reviewer GYCX
Transferability
Response to e3YZ
Thanks
Response to Reviewer GYCX
comment: Thank you for addressing the concerns raised in the review. Your additional details regarding the computational efficiency and scalability of the UrbanKGent family are appreciated. The reported latencies and the application demonstrate a significant improvement in performance, particularly when dealing with larger datasets like NYC-Large and CHI-Large.

Thank you again for your efforts to improve the paper. These additional clarifications will undoubtedly enhance the quality of the manuscript.
Thanks for your responses. MY concerns are addressed. I will raise my score to accept.
Dear reviewer GYCX:

We sincerely appreciate precious review time and valuable comments. We have carefully considered your comments and have provided corresponding responses, which we believe have covered your concerns. We hope to further discuss with you whether or not your concerns have been addressed. Please let us know if you still have any unclear parts of our work.

Best,

NeurIPS 2024 Conference Submission 16941 Authors
In this case the the transferability is indeed discussed. I have raised my rating.
Dear Reviewer e3YZ:

We are pleased that the previous rebuttal clarified most of the confusion. However, we apologize for the remaining confusion regarding "the transferability of the proposed framework in cities that are not fine-tuned." Please check the following response for a more detailed explanation.

We agree with the reviewer that it is crucial to assess whether the proposed UrbanKGent framework can enhance UrbanKG construction in cities where the data was not used for fine-tuning. In fact, we have discussed this scenario in our main results. Specifically, we **directly apply the proposed UrbanKGent framework (i.e., UrbanKGent Inference in Table 3)** to different LLMs without fine-tuning, and observe the performance changes compared to prompting LLMs with zero-shot reasoning or in-context learning paradgim.

As can be seen in Table 3, the proposed UrbanKGent framework was applied to the NYC and CHI datasets **without any fine-tuning of the LLMs for these specific cities.** Despite the absence of fine-tuning, the UrbanKGent framework could significantly improve the performance of various LLMs (including open-source models like Llama and Mistral, as well as API-based models like GPT-3.5 and GPT-4) on UrbanKGC tasks. For example, after applying the UrbanKGent framework, Llama-2-13B's RTE performance improved by 84.21%, rising from 0.19 to 0.35 under human evaluation, compared to the zero-shot reasoning paradigm. Similarly, Mistral-2-7B and GPT-3.5 showed improvements of 64.71% (0.17→0.28) and 38.71% (0.31→0.43), respectively. The improvements in KGC tasks are also substantial. These results demonstrate that our **proposed UrbanKGC framework can significantly enhance LLM performance on UrbanKGC tasks, even without city-specific data for fine-tuning**.

We will include a more detailed description of the experimental settings and results analysis in the final version of our paper to avoid such confusion. We sincerely appreciate your insightful question and the opportunity to clarify this aspect of our work.

Best,

NeurIPS 2024 Conference Submission 16941 Authors
I would like to thank the authors for the response. I would like to keep my rating as I find the argument about transferability is not that convincing.
Dear Reviewer GYCX:

We are pleased that most of the confusion has been clarified by the previous Rebuttal. We appreciate your expertise and insights in helping us improve the quality of our paper. But we apologize for the difficulty the reviewer still experienced in understanding the necessity of using LLMs for geospatial relation prediction. Please check the following response for a more detailed explanation.

We propose using LLMs for geospatial relation prediction for two primary reasons. First, existing GIS tools may struggle to extract certain spatial relations when geospatial information is incomplete. For instance, if the polygon data (i.e., the latitude and longitude boundaries) of two urban entities, such as Queens and Staten Island, is incomplete, traditional methods may fail to predict missing geospatial relations (e.g., whether they are disconnected). In contrast, LLMs can leverage both geospatial and semantic information to infer these relations. For example, an LLM might successfully infer that "Queens and Staten Island are geospatially disconnected" by directly using their semantics. Second, predicting new spatial relations often requires using multiple GIS tools or even developing new ones, which can be labor-intensive. LLMs, however, can efficiently manage this process by automatically routing tasks to existing tools or implicitly building a neural inference function for geospatial relation prediction. In our framework, we have designed the LLM to invoke various external tools to derive urban relations, and we are working to integrate more spatiotemporal tools to enhance the framework's effectiveness.

Overall, the usage of LLMs for geospatial relation prediction has practical potential and has been explored in prior research. For example, recent studies [1-3] have quantitatively evaluated LLMs' ability to predict spatial relationships and perform certain spatial calculations. Furthermore, works like CityGPT [4], CityBench [5], and BB-GeoGPT [6] demonstrate the potential of LLMs in automating complex geospatial reasoning tasks, including geospatial relation prediction. We believe these efforts are crucial for the future deployment of LLM-based applications in urban and GIS contexts. In such a scenario, we present the first attempt to use LLMs for urban geospatial relation completion within the UrbanKG construction process. We believe our approach can serve as a valuable reference for researchers in this field.

We will include the discussion on the necessity of using LLMs for geospatial relation completion in the final version of our paper. We sincerely appreciate your insightful question and the opportunity to clarify this aspect of our work.

Best,

NeurIPS 2024 Conference Submission 16941 Authors


Reference:

[1] Li, et al. “GeoLM: Empowering Language Models for Geospatially Grounded Language Understanding.” EMNLP. 2023.  

[2] Bhandari, et al. “Are Large Language Models Geospatially Knowledgeable?” ICAGIS. 2023. 

[3] Mooney, et al. “Towards Understanding the Geospatial Skills of ChatGPT: Taking a Geographic Information Systems (GIS) Exam.” SIGSPATIAL. 2023. 

[4] Feng, Jie, et al. "CityGPT: Empowering Urban Spatial Cognition of Large Language Models." *arXiv preprint arXiv:2406.13948* (2024). 

[5] Feng, Jie, et al. "CityBench: Evaluating the Capabilities of Large Language Model as World Model." *arXiv preprint arXiv:2406.13945* (2024). 

[6] Zhang, Yifan, et al. "BB-GeoGPT: A framework for learning a large language model for geographic information science." *Information Processing & Management* 61.5 (2024): 103808.
Thanks for the responses, which partially addressed my concerns except for the necessity of using LLM for geospatial relation prediction. Thus, I raised my score to Borderline acceptance.