UrbanKGent: A Unified Large Language Model Agent Framework for Urban Knowledge Graph Construction
Authors: Yansong Ning
Hao Liu
Venue: NeurIPS 2024 poster (2024)
Keywords: urban knowledge graph
knowledge graph construction
large language model agent
Primary area: natural_language_processing
TL;DR: In this paper proposed UrbanKGent, the first UrbanKG construction agent framework with Large Language Models.

Abstract: Urban knowledge graph has recently worked as an emerging building block to distill critical knowledge from multi-sourced urban data for diverse urban application scenarios. Despite its promising benefits, urban knowledge graph construction (UrbanKGC) still heavily relies on manual effort, hindering its potential advancement. This paper presents UrbanKGent, a unified large language model agent framework, for urban knowledge graph construction. Specifically, we first construct the knowledgeable instruction set for UrbanKGC tasks (such as relational triplet extraction and knowledge graph completion) via heterogeneity-aware and geospatial-infused instruction generation. Moreover, we propose a tool-augmented iterative trajectory refinement module to enhance and refine the trajectories distilled from GPT-4. Through hybrid instruction fine-tuning with augmented trajectories on Llama 2 and Llama 3 family, we obtain UrbanKGC agent family, consisting of UrbanKGent-7/8/13B version. We perform a comprehensive evaluation on two real-world datasets using both human and GPT-4 self-evaluation. The experimental results demonstrate that UrbanKGent family can not only significantly outperform 31 baselines in UrbanKGC tasks, but also surpass the state-of-the-art LLM, GPT-4, by more than 10% with approximately 20 times lower cost. Compared with the existing benchmark, the UrbanKGent family could help construct an UrbanKG with hundreds of times richer relationships using only one-fifth of the data. Our data and code are available at https://github.com/usail-hkust/UrbanKGent.

UrbanKGent: A Unified Large Language Model Agent Framework for Urban Knowledge Graph Construction
Yansong Ning', Hao Liu
AI Thrust, The Hong Kong University of Science and Technology (Guangzhou) 2 CSEE The Hong Kong University of Science and Technology
Uming0922cconnecee wwust-uss. ew..uu...uuude.................. on liuh@ust.hk
Abstract
Urban knowledge graph has recently worked as an emerging building block dis- till critical knowledge from multi-sourced urban data for diverse urban application scenarios. Despite its promising benefits, urban knowledge graph construction (UrbanKGC) still heavily relies on manual effort, hindering its potential advance- ment. This paper presents UrbanKGent, unified large language model agent framework, for urban knowledge graph construction. Specifically, we first con- struct the knowledgeable instruction set for UrbanKGC tasks (such as relational triplet extraction and knowledge graph completion) via heterogeneity-awwe aware and geospatial-infused instruction generation. Moreover, we propose tool-augmented iterative trajectory refinement module enhance and refine the trajectories distilled from GPT-4. Through hybrid instruction fine-tume finettunng with augmented trajectories on Llama 2 and Llama 3 family, obtain UrbanKGC agent family consisting of GennnK//n-/////11 version. We perform a comprehensive evaluation on two real-world datasets using both human and GPT-4 self-evaluation. The experimental results demonstrate that UrhanKGenn family can not only significantly outperform 31 baselines in UrbanKGC tasks, but also surpass the state-of-the-an L.M,, GPT-4, by more than 10% with approximately 20 times lower cost. Compared with the existing benchmark, the UrbanKGent family could help construct an UrbanKG with hundreds of times richer relationships using only one-fifth of the data. Our data and code are available at http:://gihubb.com/ssss wwwps//ooooossaaloouuuuuaa hhwkstttrbanKaC/
Introduction
Urban Knowledge Graph (UrbanKG) aims to model intricate relationships and semantics within city by extracting and organizing urban entities (e.g., POIs, road networks, etc.) into multi- relational heterogeneous graph [ As an emerging building block, multi-sourced urban data are widely used construct UrbanKG provide critical knowledge for various knowledge-enhanced urban downstream tasks, such as traffic management, pollution monitoring, and emergency response 2 5] UrbanKG has gradually become an essential tool of the modern smart city.
In prior literature, many efforts have been devoted to urban knowledge graph construction (Ur- banKGC) using massive urban data sources. In particular, one commonly used approach (6] 7 g is to extract entities from structured urban data (e.g., geographic data, city sensor data, and traffic data) and define the relationships between obtained urban entities based on manually designed rules or patterns. However, these approaches suffer heavy reliance on a deep understanding of the application domain and are labor-intensive. Recently, inspired by the success of the Large Language Models (LLMs)
Corresppnndng author.
/ntops:/huuggnngaacccom coi.uaal Akkust/UrbanKGeet-7B https//huuginnffacccousssl httpr//nne...ctoooosaatattttt/aa www.s//uaaunGnnneeettteen and httpss//huggingfacecom co/usaii hkust/UrbanKGGent 13B
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

in knowledge graph construction p TO m.
Coluabia university con) is the addect Instatation or higher abcation Now of iristry church in rasaattar.
Gecazatially
the LLMs have been adopted to improve Ur-
discosenctdd
banKGC. For instance, GeoL [12 is pre-
selatioeal Fripler ietroretas
Unbon consledge Sreph conslattin
trained on crowdsourced geographical corpus for geospatial entity recognition and relation ex-
Givvr two urben aniitite::
extnact the orter re relation))
undveretty, 1se, Sags supdine state aucldding Please complete the geopettia (nlationuhtip
the antity,
recutee with
traction. K2 [13 retrains Llama-2-7B model on manually processed and filtered geoscience text corpus for geospatial relation extraction. Never- theless, these works rely on high quality corpus annotation and computational extensive model retraining, which may discourage researchers from adopting UrbanKG for their own work.
relation, tall wetty freest.
<au, extabilsted in, 3754 scu Lacate-sr, ir, Indedry Cuuets sou, Locate-ds, Sr, now vort clop
soery, 1016 hand deciid
3
paoppatta Edstance calealation.
ectract traptet frem sonsial matatien apectrias for in relation sone retenane choad antity, recation, tall
goo-basst ancretig, to help
C diatence betueen
State Dutlding" the tereisse, inturrstty do not share the sure beandary. these tuo cetitiie
Vetiity reneal.
utesilafer-nn, 1754
LLM agent 14 15| has recently emerged c and shown remarkable zero-shot capability for autonomous domain-specific task completion. For example, Voyager [16] is a LLM-powered agent for zero-shot game exploration without re-training, and LLMLight [17] is a traffic sig- nal control agent with zero-shot LLM reasoning ability. These studies motivate us to construct tailored LLM agents to address the aforemen- tioned limitations in UrbanKGi construction.
Locate in, Aaturiters
(0), Locate In, sox York City
Lark of seterepeanne relatioesklo unkertaaniing shility
D1 Lack oF trespatial compurting Ability
Figure 1: Illustrative example of urban relational triplet extraction and knowledge graph completion. (a) The heterogeneous relationship understanding limitation of L.LMs can be addressed by injecting prior urban knowledge into instruction. (b) The geospatial computing limitation of LLMs can be alleviated by invoking external geospatial tools.
In fact, constructing an LLM agent compatible with various UrbanKGC tasks is non-trivial problem due to the following two challenges: (1) Challenges I: How to adapt LLMs for UrbanKGC? LLMs may not align well with the specific task due to the gap [1] between the natural language processing corpus for training LLMs and the domain-specific corpus in urban domain [9]]. For example, the urban text data is usually heterogeneous and contains multifaceted urban knowledge (e.g., spatial, temporal, and functional aspects) [13]. As shown in Figure Ia), the text description of "Columbia University" reflects its geographic spatial locations (i.c., spatial relationship), construction timelines (i.c.. temporal relationship), and how it provides educational service for the city (i.e., functional relationship). LLMs may require tailored aligmment to understand heterogeneous urban relationships to extract these urban spatial, temporal, and functional relations accurately. (2) Challenges 2: How to improve the capacity of LLMs for UrbanKGC? The effectiveness of LLMs for urban knowledge graph construction is restricted by their feeble numerical computation capacity 20 21 leading to their disability in complex geospatial relationship extraction 22 23)
However, the urban geospatial relationship plays a vital role in urban semantic modeling 12 and
has been widely incorporated in previous UrbanKGs g 24 As can be seen in Figure I extracting "disconnected" relation between the geo-entity "Columbia University" and "Empire State Building is useful for urban geo-semantic modeling. Accurately extracting such geospatial relationship demands necessary geospatial computing (c.g., utilizing latitude and longitude for distance calculation) and reasoning (i.e., deriving calculation results for geospatial relation reasoning) capabilities. It is appealing to improve the geospatial computing and reasoning ability of LLMs to satisfy the UrbanKGC task requirement.
To address the aforementioned challenges, in this study, we propose UrbanKGent, a unified LLM agent framework for automatic UrbanKG construction. Figure 2 illustrates the overview of Ur- banKGent. For given city, we first generate a knowledgeable instruction set for UrbanKGC tasks (relational triplet extraction and knowledge graph completion) from urban geographic and text data sources. By heterogeneity-awaan and geospatial-infusee instruction generation. as shown in Figure [Ta,, various urban spatiotemporal relationship knowledge can be encoded into the instruction, which facilitates alignment between LLMs with the target UrbanKGC tasks. Moreover, we propose a tool-augmented iterative trajectory refinement module to enhance and refine the trajectory derived by distilling GPT-4 with the above constructed instructions. Based on geospatial tool augmentation and self-refinement, the deficiency of LLMs in geospatial computing and reasoning could be alleviated, and unfaithful trajectories could be filtered out. Finally, we perform hybrid instruction fine-tuning based on the enhanced and refined trajectories on Llama 2-7/13B and Llama 3-8B variants [25] by using LoRA [26). The obtained UrbanKGent agent including 7/8/13B version, is feasible for completing multiple UrbanKGC tasks cost-effectively without extra GPT-API cost.
2

---

We conduct comprehensive experiments on two UrbanKGC tasks in two metropolises (New York City and Chicago) using both human eval- uation and GPT-4-based self-evaluation. The empirical results validate the effectiveness of the proposed LLM agent for completing various UrbanKGC tasks. Moreover, the obtained Ur- banKGent family (7/8/13B version) could help extract the same scale of triplets and entities of existing UrbanKG benchmark [8] using only
Relational Tripict Extraction
Knowledge Graph Completion
Inderwnce UrbanKOG Tacks
Knowledgabbic Instraction Generation
Tool- Toul-Augmented lterativv
Hybrid instruction
Trajentoy Refluement
Tuuinn
UrbamiGC Agent Construction
AOI
Road Network
POt
Review
Web Page
Liram Grographic.or Data Scurres
one-fifth of data, and even expand the types of relations by hundreds of times.
Figure 2:
The framework of UrbanKGent.
Our contributions are summarized follows: (1) We propose the first UrbanKGC agent framework UrbanK Gent and UrbanKGent family to provide real-world UrbanKGC service, offering new oppor- tunities to advance UrbanKG studies. (2) We propose knowledgeable instruction generation module and tool-augmented iterative trajectory refinement method, which align LLMs to UrbanKGC tasks and compensate for their geospatial computing and reasoning inability. (3) Extensive experiments on two real-world datasets validate the effectiveness of proposed framework and uncover exceptional performance across UrbanKGC tasks.
UrbanKGC Data Description
2.1 Data Collection
We first acquire urban knowledge for two large cities New York City and Chicago from two data sources. Table summarizes the statistics of the raw datasets.
2.1.1 Geographic Data
The geographic data provides critical urban spatial structure information and functional semanties, which has been widely used in previous UrbanKG studies [8] [4 27 28).
Area-Of-Interst (AOI) Data. AOI data describes the urban spatial area structure, including urban commercial areas (e.g., shopping centers), residential areas (e.g., communities), and so on. In this work, we first follow UUKG [ to acquire the AOI name and geometry value from NYC Gov and CHI Gov | Next, we use the AOI name to search their text description from Wikipedia and C4 dataset. Each AOI record contains an AOI name, polygon geometry value, and text description. For example, ['Jamaica Bay", polygon (-73.86 40.58, ...). "Jamaica Bay is an estuary ...") "/ is the record of the AOI "Jamaica Bay" with geometry value and text description.
Road Network Data. Road data describes the urban spatial network, including urban motorways, overpasses, and so on. We first follow [8] to obtain the road name and geometry value from Open Street Map (OSM M Then, following the same text acquisition operation in AOI data, we crawl the textual description of each road record from Wikipedia. Each road record contains road name, a linestring geometry value, a road type and text description. For example, ("Central Park Avenue", linestring (-73.87 40.90 ...), primary, "Central Park Avenue is boulevard in ... "J describes the primary road named "Central Park Avenue" with a linestring geometry value and its textual description.
Point-Of-Interest (POI) Data. POI data represents different urban functions (c.g., residential and commercial), which have been widely adopted in many recent UrbanKG works 24 6 We first follow [ to obtain the POI name, and geometry value from OSM. Then the textual description of each POI record could be crawled from Wikipedia following the similar process. Each POI record contains POI name, coordinate geometry, a POI type, and a text description. For example,
https://www.nyc.ggom
https//www.chicago.ggg
https//huggingfacc.ooddaat coldanasets/allennaiicc
http://www..oeensuuuumnnogggg
3

---

[ ("Trump World Tower", coordinate (-73.96 40.75), residential, "Trump World Tower is residential condominium "I is the record of the POI "Trump World Tower".
Table 1: The statistics of raw datasets.
Dataset
Description
New York City
Chicago
AOT
192
Geographic Data
of read
6,765
2,241
5,872
5,877
2.1.2 Text Data
of review
16,360
13,627
Text Data
web page
11,596
7,283
The text data provides rich contextual knowl- edge of the city space from different perspec- tives (e.g., the spatial context) [13] and it plays an important role in geospatial understanding. this work, we collect two types of text corpus.
Review Data. The review of urban places provides commercial information that citizens use to make business decisions [29 1 review data from Google Map Specifically,, we first manually split the city into multiple rectangular regions, then we utilize the Google Map API to query the places contained within each region and their reviews. Each review record contains place name, a coordinate geometry value, a rating, and a text review. For example, [ ["Lifestyles Academy Inc", coordinate (-87.87 41.65), 4.9, "Very nice organization and ... "J is the review record of place "Lifestyles Academy Inc".
playing critical role in urban knowledge distillation. We collect
Web Page Data. The web page data works as the general text corpus for the city, and contains rich geoscience knowledge that has been utilized in recent urban entity and relation extraction studies [13]. We collect web page data from the Google search engine. Specifically, we first input the name of the crawled AOI, Road, and POI record into Google. Then concatenate the textual sentences of the top 10 retrieved web pages, Each web page record contains a long urban text description.
2.2 Data Preprocessing
Before constructing the UrbanKGC dataset, we first preprocess the raw datasets. We filter out AOIs, roads, POIs, reviews, and web pages whose crawled textual descriptions are null value, short (e.g.. less than ten word description) or meaningless (e.g., just repeating the POI name). In addition, we remove irrelevant information from the text description, such as non-English characters, non-ASCII gibberish, website addresses, and so on. More details can be found in Appendix A
3
Preliminary
This section presents the UrbanKGC task definition and provides task analysis.
3.1 Task Definition and Problem Formulation
Before diving into the technical details, we first introduce the definition of UrbanKG:
Definition UrbanKG. The UrbanKG is defined as a mulii-rlational graph G (E, R, R, where E. R and F is the set of urban entities, relations and facts, respectively. In particular, facts are defined as F = {(h, r, h, t E.r E RJ, where each triplet (h, t) describes head entity is connected with tail entiry via relation
The UrbanKG encodes diverse urban semantic knowledge by connecting urban entities into multi- relational graph. This work aims to construct an UrbanKG from collected unstructured text data. We decompose the UrbanKG construction ( (UrbanKGC) process into two sequential knowledge graph construction tasks, namely relational triple extraction (10] and knowledge graph completion [II]. We first provide the basic definition for these two subtasks, and then introduce the problem formulation of this work.
3.1.1 Task Definition
Relational Triplet Extraction (RTE). Given the unstructured texts, this task achieves joint extraction of entities and their relations [10 which are in the form of triplet (h, r. f). For instance, given the
hatps//www.gogge..oommmas

---

urban text sentence "Columbia Uni- versity is private Ivy league re- search university in New York City.", this task aims to identify two entities "Columbia University" and "New York City" and their relation "locate-in", ",n"""" de- scribed as triplet < <Columbia Univer- sity, locate-in, New York City>.
staeeill weng
THI Wea
04
D3
00
PD
groppadd sdations
Figure 3: Quantitative performance analysis of prompting
Knowledge Graph Completion (KGC). Given head entity and a tail entity t, this task is to predict the missing relation between them [IT]. and the tail entity "Empire State Building", this task is to predict that their missing relation, e.g.. "disconnected", described as triplet <Columbia University, disconnected, Empire State Building>.
GPT-4 for UrbanKGC tasks. The result obtained by com- paring 50 GPT-4's outputs with the human's 's annotation.
For instance, given the head entity "Columbia University"
3.1.2 Problem Formulation
Given the urban unstructured text data, the desired output is an UrbanKG G. In this paper, this problem is decomposed into two sequential subtasks: (1) Relational Triplet Extraction: the first task extracts relational triplet (h, r, t) from the urban text data. The output of RTE task is 91 =(E, R1, where & and is the set of extracted entities and relations, while Fi is the set of extracted triplets. (2) Knowledge Graph Completion: for the given head entity h and tail entity & in 91 the second task is to predict the geospatial relationship" between them. The output of this task is G2 =(E, R2, F2), where R2 and F2 is the set of completed relations and triplets. By sequentially completing the above two tasks, we can obtain the constructed UrbanKG G (E. R, R2, Fi F2)
3.2 Quantitative Task Analysis
As shown in Figure I we qualitatively find that LLMs lack urban heterogeneous relationship understanding ability and experience in geospatial computing and reasoning difficulty when adopting it for UrbanKGC tasks. This subsection presents quantitative analysis of these two challenges.
Heterogenous Relationship Understanding, The ability to understand heterogeneous relationships is ubiquitous in distilling knowledge from the massive urban corpus. For example, the text description in Figure T illustrates a place from spatial location, temporal time, and functional aspects. Capturing these heterogeneous semantics is important for urban knowledge distillation. We perform quantitative analysis by randomly sampling 50 urban text data and then prompt GPT-4 to complete relational triplet extraction by providing only the basic task description. As shown in Figure [3] [), find the LLMs experience serious misjudgment (i.e., extract wrong triplets or miss the triplet) on urban spatial, temporal, and functional triplet extraction. This indicates the limited capacity of LLMs to understand heterogeneous relationships.
Geospatial Computing and Reasoning, Geospatial computing and reasoning techniques are widely used in many previous UrbanKG studies [8] 24] for urban geospatial relation extraction. In recent works 23 311, the geospatial skills of LLMs have also been demonstrated to lack geospatial awareness and reasoning ability (22). To identify potential limitations, we quantitatively investigate how LLMs can perform on geospatial relation completion tasks. Specifically, we construct 100 head and tail entity pairs, covering five geospatial relations in the KGC task, and then prompt GPT-4 to predict with basic task description and geospatial relation candidates. As shown in Figure (b), find that GPT-4 performs poorly on five geospatial relation completion. This further validates the disability of LLMs in geospatial computing and reasoning.
b)), we
4 UrbanKGC Agent Construction
This section presents the proposed UrbanK agent construction framework.
*We follow GeoLM 12 to provide five RCC relationship 30 candidates: Disconnection (DC), external
connection (EC), equality (EQ), partial overlap (PO), and tangential and non tongennttt proper parts (IN). Details are in Appendix A
5

---

Krewindge Grapn
odivee arian test.
Glven aram teats rixa readills
Tho dialng
srofile
alliwe
csitiiy deffection
<Task profille detinittase
Aspweted
Tragectary
Ferine
Trigectary Verffeer
RerSand Irajecterden des
UrbenadC agent
eolaticnal tesplet sutractiee
oslven araan teats
con Let's think by stop
Exterall seceuatdal teols
oqartis sittiry and relation tygecy
Distanee
within
(007> thant
Spocial vay
Toqpraal Virw
within Palyggn
telattneal Extraction
Complation
functiooal
(a) Konalndgmable Instruction Generation
(b) Tool augeented Iterative Trajectory Refinement
Hybrid Instruction
Fine-
Figure 4: An overview of UrbanKGent Construction.
4.1 Overview
The overall pipeline of the UrbanKGent framework illustrated in Figure 4 (1) Knowledgeable Instruction Generation consists of the heterogeneity-aware and geospatial-infused instruction gen- eration modules for aligning LLMs to UrbanKGC tasks. (2) Tool-augmented Iterative Trajectory Refinement proposes geospatial tool interface invocation and iterative self-refinement mechanisms to enhance and refine generated trajectory. (3) Hybrid Instruction Fine- tinettunigg fine-tune LLMs based on the refined trajectories for cost-effectively completing diverse UrbanKGC tasks.
4.2 Knowledgeable Instruction Generation
We first construct the knowledgeable instruction to adopt I.LMs for two UrbanKGC tasks, including relational triplet extraction (RTH) and knowledge graph completion (KGC). Figure 4 a) illustrates the overview of the instruction construction process of these two tasks.
Heterogeneity--warr Instruction Generation for Relational Triplet Extraction. As discussed in Section 3 the urban text contains diverse heterogeneous relationships, thus we consider multiple views with both urban entity and relation definition for relational triplet extraction. In particular, we construct multi-view instruction template for the urban relational triplet extraction, including spatial view, temporal view, and functional view. Each view is a multi-turn question-ansee dialog [32 consisting of entity recognition, relation extraction, and triplet extraction module.
For the spatial view, we devise a two-turn dialog to align LLMS for spatial triplet extraction. In the first turn, we inject spatial entity and relation definition into the instruction template to guide LLMs to understand spatial characteristics and then extract potential spatial entities (c.g., University) and relations types (e.g., locate-in). In the second turn, the extracted types are explicitly fed into the instruction template for spatial triplet extraction. Intuitively, the spatial view allocates dedicated urban knowledge for LLMs extract urban spatial relationships. Similarly, construct the temporal view and functional view for corresponding temporal and functional triplet extraction, independently.
-ooopattallinfused infused Instruction Generation for Knowledge Graph Completion. Despite heterogeneity-awee aware instruction enabling I.MMs to extract urban triplets from various perspectives, the geospatial relationship between geospatial entities cannot be directly extracted. Therefore, we introduce geospatial-infused instruction generation module guide L.LMs to complete missing geospatial relationships.
Geospatial
First, we incorporate the geometry information (i.e., the latitude and longitude) of geo-entities into instruction, so that the LLMs can utilize these geospatial values for relation inference. Second, we add the geospatial relationship definition to the instruction to guide LLMs in understanding the geospatial relationship definition. Intuitively, LLMs can refer to geospatial knowledge and make practical solutions for the KGC task. We provide the detailed instruction template in the Appendix B
4.3
Tool-augmented Iterative Trajectory Refinement
4.3.1
Trajectory Generation
With the initial UrbanKGC instructions constructed, the following step is to generate reasoning trajectories (3), which will be used fine-tune L.LMs tailored to UrbanKGC task. Specifically, we follow FireAct (34] and use Chain-of-Thouge (CoT) (35), gradient-free frre technique, to prompt

---

GPT-4 (i.e., add prompt trigger "Let's 's think step by step" at the end of RTE and KGC instructions template) generate the reasoning trajectories for UrbanKGC tasks.
The generated CoT trajectories could provide a step-by-step reasoning solution for UrbanKGC tasks. Nevertheless, the complex geospatial relationships cannot be easily extracted discussed in Section 3 and recent geospatial reasoning works [31 23 [22]. Therefore, we introduce a tool invocation module guide LLMs invoke tailored external geospatial tools [36] to enhance their geospatial computing and reasoning capacity for UrbanKGC tasks.
4.3.2 Tool Invocation for Trajectory Augmentation
We conduct two sequential procedures: tool invocation for geospatial computing support and trajectory deliberation for reasoning enhancement.
Tool Invocation. First, we construct a geospatial reasoning toolkit (e.g. distance calculation, eight interfaces in total shown in T) by prompting GPT-4 for self- seff-programming Then, we construct tailored prompts to guide L.MSs to invoke these interfaces. Specifically, the prompt concatenated with an illustrative description of the function of each geospatial tool and a task instruction (i.e., "Which types of tool interface you need"). Intuitively, the external tool allocates calculation results for LLMs to infer missing geospatial relation. The toolkit description can be found in Appendix C
Trajectory Deliberation. After manipulation with external tools, we prompt LLMs to refine uncertain reasoning steps based on these obtained manipulation results. Specifically, we construct the prompt by concatenating with manipulation results (e.g., the distance and geohash value of geo-entity) and task instruction (i.e.. "Please refine your reasoning process"). After feeding the prompt into GPT-4, the enhanced trajectory obtained. Detailed prompt information can be found Appendix C
4.3.3 Iterative Trajectory Self-refinement
Despite tool-augmented deliberation improving the geospatial computing and reasoning ability of LLMs, enhanced trajectories may not all be faithful [37]. To alleviate potential error and ensure the trajectory quality (38), we refine these trajectories via an iterative self-refinement mechanism
39). Specifically, we iterate two sequential blocks: (i) Trajectory verifier: given the trajectory, the verifier aims to provide feedback for refining the reasoning process; (ii) Trajectory updater: given the trajectory and feedback, the updater will further refine the current trajectory based on the feedback.
Trajectory Verifier.
We construct tailored prompt to ask LLMs to generate feedback. Specifically, we use a simple but effective trigger (" ("Judge whether all extracted triplets are correct and provide improvement suggestion") ') to prompt LLMs to provide feedback. If the trajectory longer requires modification, we let LLMs respond with "This is a faithful trajectory". Such a verification step lets LLMs make reflections and improve the correctness of the trajectory.
Trajectory Updater. The updater utilizes provided feedback to refine the current trajectory via prompt trigger "Follow suggestion to refine the reasoning process". Intuitively, the feedback may address multiple aspects (c.g., missed triplet in the RTE task or unfaithful reasoning process in the KGC task) of the unfaithful trajectories.
We iterate the trajectory verifier and updater until the predefined stopping condition is satisfied. The stopping condition is determined by either meeting the maximum number of iterations (we set it at three to avoid excessive cost) or when the verifirr confirms all trajectories are faithful. Upon meeting the stopping condition, we use the last refined trajectory for further fine-tuning. Detailed prompt information can be found in Appendix C
4.4 Hybrid Instruction Fine-Tunie Fine-Tuning
To construct cost-effective UrbanKGC agent, we further utilize trajectories (generated by GPT-4) to fine-tune a smaller open-source LLM for faster inference speed and lower cost (i.e., prompting GPT-4 for UrbanKGC is expensive). Specifically, finetune the LLM via the mixed-task instruction-tuning strategy B3 The goal is to enhance the LLMs' capabilities in diverse UrbanKGC tasks.
Mixture Training, Set the base language model as M, and PM (y | x)) represents the probability distribution of response y given instruction J. We consider the trajectory set on two UrbanKGC tasks, i.c., DRTE and D DKGC. Since both the instruction and the target output are formatted in natural
7

---

Table 2: The statistics of constructed UrbanKGC dataset.
Dataset
NYC-Instruct
NYC
NYC-Large
CHI-Instruct
CHI
CHI-Large
RTR
232
2,089
40,480
122
1,102
28,868
Records
KGC
232
2,080
33,534
122
1,101
28,607
language, we can unify the training into an end-to-end sequence-to-sequene way. Formally, the optimization process aims to minimize the loss of language model .M as follows:
L Dain [log PM (y x)] + ((e)))>Doc [log PMggMM [=]]
(1)
where x and y represent the instruction input and instruction output in the trajectory, respectively.
Training Setup. We choose the chat version of open-sourced Llama 2-7/13B and Llama-3-8B as our backbone models, and fine-tune Llama using LoRA strategy [26].
4.5 Inference on UrbanKGC Task
Via hybrid instruction ine-ttunig. the obtained LLM UrbanKGent can be trained to follow the instructions to finish the UrbanKGC task. We prompt UrbanKGent to complete UrbanKGC tasks by following the pipeline shown in Figure For the RTE task, we sequentially execute entity recognition, relation extraction, and relational triplet instruction generation, iterative aelf-refinement and output the extracted triplets. For the KGC task, we sequentially execute KGC instruction generation, external tool augmentation, iterative self-refinement block, and finally output the completed triplets.
Experiments
5.1 Experimental Settings
Dataset. In this work, two sequential tasks (i.e., RTE and KGC) of UrbanKGC are within an open- world setting (i.e., predefined ontology) [40 41 We construct the RTE and KGC datasets of NYC and CHI by sampling uniformly from five raw data in Table [ respectively. As shown in Table 2 we first construct two small datasets (i.e,, NYC-Instruet and CHI- CHI-Instruct) for instruction fine-tuning and two middle datasets (i.e., NYC and CHI) to validate the performance of the constructed UrbanKGC agent. The remaining data works the large-scale UrbanKGC dataset (i.e., NYC-Large and CHI- Large) in real-world scenarios shown Ta The three types of datasets are non-overlapping to prevent data leakage. More dataset construction details are in Appendix A
Baseline Methods. We provide a comprehensive comparison of our method with existing paradigm gm: (1) End-to-end Models: For the zero-shot RTE task, utilize the end-to-end generation model RelationPrompt [42] and PRGC [10]. For the KGC task, we fine-tune KG-BERT 43 and KG- T5 [4] with the QA pairs constructed from the self-instruct dataset. (2) LLMs-based Zero-shot Reasoning [45] We directly prompt the LLMs with basic task definitions to get the answer without training. (3) LLMs-based In-context Learning [35]: We sample 3-shot QA pairs as demonstrations from the self-instruct dataset as examples and get the answers from the L.MMs without training. (4) Vanilla Fine-tuning [11] We directly fine-tune the L.MMs using the QA pairs constructed from the self-instruct dataset, and then prompt the LLMs with basic task definition without demonstrations. (5)
UrbanKGent Inference: We directly prompt the LLMs using the UrbanKGgent inference pipeline
in Section 4.5 The prompt templates of the above baseline methods are in Appendix B
Implementation and Detail Settings. In experiment, select Vicuna [46), Alpaca 47]. Mistrial [48], Llama-2 [49]. Llama-3 [50], GPT-3.5 [5T] and GPT-4 [51] as the backbone LLM M.
All experiments conducted eight NVIDIA A800 GPUs. For the GPT-3.5 and GPT-4, adopt API and 1pt-4-0661 API.
the gpt-3. 4pt-3.5-ttrbo-16k-
Evaluation Protocol. are not visible, the classical metric (e.g., and Hits@ 10) is not applicable. In this work, we regard evaluation as the binary classification, i.c., if the extracted triplet in RTE task is correct and if the completed relation in KGC task is correct. We follow recent LLMs--ased KGC works [ to employ accuracy an evaluation metric. To make a comprehensive evaluation of the experimental results, we employ both of the human evaluation and GPT evaluation, which has been widely used in many LLM studies [52 45
Since UrbanKGC tasks in this work follow ooen-world setting where labels
For Human Evaluation, we employ human annotators to evaluate the results
8

---

Table 3: The main results of relational triplet extraction (RTE) and knowledge graph completion (KGC). We report the accuracy (acc) and confidence for GPT evaluation on two datasets, and report accuracy (acc) for the Human evaluation approach. The best baseline performance is underlined.
NYC
CHI
Type
Models
GPT (acclcunfidence)
Human (ace)
GPT (aco/confidence)
Human (ace)
RTE
KGC
RTE
KGC
RTE
KGC
RTE
KGC
KG-BERT
0.24/3.15
0.23
0.19/4 12
0.24
End-to-ond Models
KG-T5
0,21/4.02
0.21
0.15/9
0.24
RelationPromp
0.12/3.38
0.12
0.21/3.53
0.18
PRGC
0,08/4.11
0.06
0.13/4.15
0.15
Vicuna-7B
0.12/2.84
0.19/4.06
0.14
0.16
0.22/4.12
0.14/403
0.21
0.18
Alpaca-7B
0/1
0.17/3.8 87
0.15
0.17
0.23/3 96
0.1644.5
0.20
0.16
Mistrll7B
0,14/4.12
0.21/4.11
0.17
0.18
0.21/3.75
0.15/3.76
0.19
0.19
Llama 2.7B
0.14/1.96
0.18/3.75
0.16
0.18
0.26/1.96 196
0.15/2.83
0.21
0.22
Zero-shot
Llama-3-8B
0.15/4.02
0.15/4.02
0.20
0.21
0,24/3.75
0,15/4.08
0.22
0,22
Reasoning
Llamm 2. 13B
0.21/2.07
0.28/3.91
0.19
0.22
0.22/2.19
0616/4447
0.22
0.24
Llama-2-70B
0,25/3.07
0.28/3.75
0.22
0.24
0,27/3.55
0,16/2.47
0.24
0,23
Llama-3-70B
0,24/4.18
0.29/4.31
0.23
0.24
0.26/3.98
0.17/4.26
0.25
0,23
GPT-3,5
0.29/4.11
0.36/3.47
0.31
0,23
0.3./3.79
0,31/3.16
0,31
0.29
GPT44
0.38/4.03
0.39/3.82
0.39/4.08
0.32/4.03
0.43
0.35
Llama 2: 2:7B
0,18/2.15
0.21/3.96
0.19
0.18
0,25/2.44
0,18/3.27
0.23
0,20
In-contexi
Llama-3-88
0.17/4.06
0.18/3.53
0.21
U.22
0,28/4.31
0.17/4.14
0.24
0.21
Learning
Llama-2-138
0.26/3.52
0.31/3.28
0.23
0,24
0,28/2.65
0.21/2.53
0,25
0,26
GPT-3.5
0.41/4.65
0.42/4.08
0. 0.36/4.24
0.36/4.23
0.39
0.36
Vanilla
1.ama-2-78
0.32/4.37
0.38/3.65
0.32
0.27
0.29/3.80
0,30/3.65
0.33
0.31
Fine luning
Llama-3-88
1.
0.35/4.18
0.35
0.26
1.31/4.18
0.29/4.15
0.32
0.34
Llama-2-138
0,35/4.26
0.41/3.92
0.39
0,29
0,31/4.14
0.293.887
0.37
0.35
Vicuma-70 7B
0.24/3.07
0.24/3.95
0.29
0.23
0.27/4,
0.22/3.95
0.23
0.25
Alpaca-7B
0.26/3.85
073/3.83
0.26
0.22
0.27/3.83
0.21/4.1
0.27
0.29
Mistral
0.26/4, 15
0,25/4.08
0.28
0.23
0.25/3.61
0.21/4.08
0.25
0.26
Llama 1lama-2-78 2-7B
0,27/3.05
0.26/4.12
0.28
0.24
0,27/2.87
0,24/3.54
0,26
0.29
UrbsnKGent
Llama-3-8B
0,29/4.15
0.31/4.08
0.33
0.26
0.26/3.22 28
0.24/3.97
0.30
0.31
Inference
Llana 2: 13B
0.31/3.87
0.32/3.56
0.35
0,27
0,28/3,24
0,26/3.28
0,31
0.32
Llama-2-770
0.33/4,28
0.35/4.27
0.33
0.29
0.29/3.80
0.28/4.01
0.32
0.34
Llama-3-70B
0,35/4.26
0.3644.81 81
0.34
0,28
0,29/4, 12
0,29/4 81
0,31
0.35
GPT.3.5
0,43/4.12
0.46/388
0.34
0,40/4.21
0.39/3.87
0.46
0.41
GPT4
0.45/4.08
0.48/4.02
0.47
0.42
0.46/4.17
0.41/4.35
0.52
0.43
UrbanKGeut-7B
1.46/4.12
0.49/3.97
0.48
0.44
0,49/4.28
0.43/4.58
0.54
0.45
t 2.222%
1 2.008%
2.08%
1.76%
1 6.52%
1 7.888
3.84%
1 1.66%
UrhanKGent-SR
0.47/3.97
1 4.44%
0.51/4.15
1 16.25%
4.26%
0.49
7.14%
0.45
0.49/3.97
6.52%
0.44/4.05
1 7.32%
5.77%
0.55
6.98%
0.46
UrbanKGent-13B
0,52/4.38
0.56/4.13
0,54
0.47
0.53/4.15
0.48/4.42
0.59
0.49
15.56%
1 14..29 29%
14.89%
t 11.90%
15.22%
17.07%%
13.46%
13.95%
200 random samples. As for the GPT Evalua- tion, we use GPT-4 to evaluate the model perfor- mance on the full data to escape intensive labor. In this work, the GPT-4's evaluation has been demonstrated be consistent with the human evaluation. Detail is in Appendix D
Table 4: Statistics comparison of constructed Ur- banKGs in New York and Chicago between Ur- bankGent and existing benchmark.
Dataset
# Entity
# Relation
# Triplet
Data Volume
NYC-Large CHI-Large
228,928
2.138 1,336
905,442 563,290
40,480
95,813
28,607
NYC-UUKG
236,287
13
930,240
236,277
5.2 Main Result
CHI-ULKG
140,602
13
564,400
140,577
The performance results are reported in T As can be seen, the constructed agent outperforms all thirty-one baseline models on two UrbanKGC datasets. Specifically, the UrbanKGent- 13B achieves (15.56% 14.29%, 14.89%, and 11. 1190%) improvements compared with the state-of-the-art GPT-4 with the same inference pipeline on NYC. The improvements on CHI are (15.22%, 17.07%, 13.46%, and 13.95%). respectively. Moreover, the UrbanKGent-77888 also achieve comparable performance compared with the GPT-4.
Meanwhile, we observe that the zero-shot LLMs perform poorly in the UrbanKGC tasks, even using GPT-4. In addition, although the demonstrations provided by In-context-learning can incorporate the Urbank task information, the performance gain is limited. Besides, we find that fine-tuning LLMs can make obvious improvements in the overall performance. Through vanilla fine-tuning, the Llama-2-7/13B and Llama-3-8B could achieve comparable performance with GPT-3.5 under the ZSL settings.
Moreover, although the various LLM backbones using the UrbanK inference pipeline perform slightly worse than the vanilla fine-tuning method, they could obtain better performance compared

---

with zero-shot reasoning and In-context leamn- ing paradigms. Such results demonstrate the benefit of knowledgeable instruction design and external tool innovation, but also indicate per- formance bottleneck. As deeper exploration, our work fills this gap through hybrid instruc- tion fine-tuning, and the fine-tuned Ur agents, whether 7B, 8B or 13B, can achieve state-of-the-ee art performance in UrbanK GC tasks. We provide an in-depth analysis of the proposed UrbanKGent framework in Appdendix E.2
@51 0LL.66672
Litudkaann 2000 COPE 333-03
(1772287
20
50
Cost (dallle)
Figure 5: The model latency and cost of con- structed UrbanKGent-Ie 13B and GPT-4 in Ur- banKGC. We report the total inference time and cost of 1,000 RTE and KGC tasks.
5.3 Agent Application
We first derive UrbanKGent-Ie 13B for initial GrbanKGs acquisition in New York City and Chicago. After proper filtering and merging of the triplets, we obtain two large-scale UrbanKGs shown in T Compared with existing UrbanKG benchmark [8], we only use roughly one-fifth of the data for constructing the UrbanKGs with the same scale of triplets and entities, and even expanding the variety of relationships to a hundred times the original types. Moreover, we also provide efficiency analysis in Figure[5] As can be seen, UrbanKGente 13B achieves lower inference speed in latency and reduce the cost by roughly 20 times in both of RTE and KGC tasks. More details is in Appendix E.
Related work
Domain-Oriented Agent Construction. The concept of language agent 34 has become very popular recently, and a variety of I.IM agents targeting different domains have heen proposed. For example, Voyager [16] is constructed for automated game exploration, WebGPT [17 is an HTML agent for diverse document understanding tasks, LLMLight (53) constructs a language agent for transportation domain, K2 [13, GeoGalactica [19] and GeoLLM 12) propose to re-train language agent for geospatial semantic understanding. In addition, many recent works like Auto-GPT [54] and CAMEL (55) aim at proposing an autonomous agent framework for agent construction. Nevertheless, there is still no UrbanKGC agent construction framework for the urban computing domain.
LLMs for Knowledge Graph Construction. Recently, the advent of L.MSs (56) invigorated the field of NLP. Many studies have begun to explore the potential of L.MMs in the domain of KG construction. For example, [32 57 finds that transforming the NER and RE task into multi-turn question-answerrng dialog could improve the model performance. 9 explicitly derive syntactic knowledge to guide LLMs think, which could develop the performance of NER. Despite these LLM-driven KG construction methods (58) 40 in general domains being widely investigated, KG construction in urban domain still remains an open challenge (59).
Urban Knowledge Graph. Urban knowledge graph has been proven useful in various urban tasks, such as traffic flow prediction [60]
61
27 62).
mobility prediction [6]. site selection (7). city profiling Their common approach involves manually extracting
[63] crime prediction and so on urban entities and defining urban relations construct urban knowledge graph. For example, (6] construct a dedicated spatiotemporal knowledge graph regarding trajectory and timestamp as entities improve trajectory prediction and (7) construct user check-in relations to help mobility prediction. Nevertheless, existing UrbanKGs heavily rely on manual design, leading to high labor costs.
g 64
65
7 Conclusion
In this work, we proposed UrbanKGent, the first automatic UrbanKG construction agent frame- work with LLMs. We first constructed knowledgeable instruction set to adopt LLMs for different UrbanKGC tasks. Then, we proposed a augl-nugeeteed iterative trajectory refinement module to facilitate the instruction tuning of various large language models. Extensive experimental results demonstrate the advancement of Urban Gent in improving UrbanKGC tasks. The obtained Ur- banK Gent agent family, consisting of 7/8/13B version, with lower latency and cost compared with derving GPT-4 for UrbanKG construction. We hope the open-source Urban GGent can foster future urban knowledge graph research and broader smart city applications.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the National Natural Science Foundation of China (Grant (20.62102) 10, No.92370204), National Key R&D Program of China (Grant No. 2023YFF0722004), Guangzhou- HKUST(GZZ) Joint Funding Program (Grant No. 2032000303000003 Education Bureau of Guangzhou Municipality,
References
[1] Weijia Zhang, Jindong Han, Zhao Xu, Hang Ni, Hao Liu, and Hui Xiong. Urban foundation models: survey. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 6633-6643. 2024.
[2] Fangzhou Sun, Abhishek Dubey, and Jules White. Dxnat-deep neural networks for explaining non- recurring traffic congestion. 2017 IEEE International Conference on Big Data (IEEE BigData 2017). Boston, MA, USA, December 11-14, 2017, pages 2141-2150, 2017.
[3] Quanjun Chen, Xuan Song, Harutoshi Yamada, and Ryosuke Shibasaki. Learning deep representation from big and heterogeneous data for traffic accident inference. In Proceedings of the Thirtieth AAA! Conference Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA, pages 338-344, 2016.
[4] Weiyu Cheng, Yanyan Shen. Yanmin Zhu, and Linpeng Huang. neural attention model urban air quality inference: Learning the weights of monitoring stations. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence. AAAI 2018, USA, February 2-7, 2018. pages 2151-2158, 2018.
[5] Ying Sun, Hengshu Zhu, Lu Wang, Le Zhang, and Hui Xiong. Large-scale online job search behaviors reveal labor market shifts amid covid-19 19, Nature Cities, 1(2):100 163, 2024.
[6] Huandong Wang, Qiaohong Yu, Liu, Depeng Jin, and Yong Li. Spatio-temporal urhan knowledge graph enabled mobility prediction. pages 18::1-18:22 2021.
[7] Yu Liu, Jingtao Ding, and Yong Li. Developing knowledge graph based system for urban computing. In Proceedings the 1st ACM SIGSPATIAL International Workshop Geospatial Knowledge Gruphs, pages 3-7, 2022.
[8] Yansong Ning, Hao Liu, Hao Wang, Zhenyu Zeng, and Hui Xiong. Uukg: unified urban knowledge graph dataset for urban spatiotemporal prediction. Advances in Neural Information Processing Systems, 36, 2023.
[9] Tingyu Xie, Qi Li, Jian Zhang, Yan Zhang. Zuozhu Liu, and Hongwei Wang. Empirical study of zero-shot ner with chatgpt. arXiv preprint arXiv:2310, 10035, 2023.
[10] Hengyi Zheng, Rui Wen, Chen, Yifan Yang. Yunyan Zhang, Ziheng Zhang, Ningyu Zhang, Bin Qin. Ming Xu, and Yefeng Zheng. Prge: Potential relation and global correspondence based joint relational triple extraction. arXiv preprint arXiv:2106. 098999 2021.
[11] Liang Yao, Jiazhen Peng, Chengsheng Mao, and Yuan Luo. Exploring large language models for knowledge graph completion. arXiv preprint arXiv:2308. 13916, 2023.
[12] Zekun Li, Wenxuan Zhou, Yao-Yi Chiang, and Muhao Chen. Geolm: Empowering language models for geospatially grounded language understanding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5227-5240, 2023.
[13] Cheng Deng, Tianhang Zhang, Zhongmou He, Qiyuan Chen, Yuanyuan Shi. Le Zhou, Luoyi Fu, Weinan Zhang, Xinbing Wang, Chenghu Zhou, et al. Learning foundation language model for geoseience knowledge understanding and utilization. arXiv preprint 74Xiv2230605006 2023.
[14] Wangchunshu Zhou et al. Agents: An open-source framework for autonomous language agents. ArXiv, abs/2309.07870, 2023.
[15] Zhiheng Xi et al. The rise and potential of large language model based agents: A survey. ArXiv, abs/2309.07864, 2023.
[16] Guanzhi Wang, Yugi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.1629), 2023,
11

---

[17] Izzeddin Gur. Hiroki Furuta, Austin Huang. Mustafa Safdari. Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and program synthesis. arXiv preprint Xix:2337.112856 2023.
[18] Yao Fu, Hao Peng, Litu Ou, Ashish Sahharwal, and Tushar Khot. Specializing smaller language models towards multi-step reasoning. arXiv preprint arXiv:2301.12722 2023.
[19] Zhouhan Lin, Cheng Deng, Le Zhou, Tianhang Zhang, Xu, Yutong Xu, Zhongmou He, Yuanyuan Shi, Beiya Dai, Yunchong Song, et al. Geogalactica: A scientific large language model in geoscience. arXiv preprint wxii:2441.0045555 2023.
[20] Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier, and Julius Berner. Mathematical capabilities of chatgpt. arXiv preprini arXiv:2301.13867 2023.
[21] Beichen Zhang, Kun Zhou, Xilin Wei, Wayne Xin Zhao, Jing Sha, Shijin Wang. and Ji-Rong Wen. Evaluating and improving aol-auumented computation-intessit math reasoning. arXiv preprint arXiv:2306.02444 2023.
[22] Prabin Bhandari. Antonios Anastasopoulos, and Dieter Pfoser. Are large language models geospatially knowledgeable? In Proceedings of the 31 ACM Internationaa Conference on Advances in Geogruphic Information Systems, pages 1-4, 2023.
[23] Peter Mooney, Wencong Cui, Boyuan Guan, and Levente Juhász. Towards understanding the geospatial skills of chatgpt: Taking geographic information systems (gis) exam. In Proceedings of the 6th ACM SIGSPATIAL. International Workshop on Al for Geographic Knowledge Discovery, pages 85-94, 2023.
[24] Yu Liu, Jingtao Ding, Yanjie Fu, and Yong Li. Urbankg: An urban knowledge graph system. ACM Transactions on Intelligent Systems and Technology, pages 1-25, 2023.
[25] Hugo Touvron, Louis Martin, and Kevin Stone al. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288 ,09288,,8888.22222 2023.
[26] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, arXiv preprint arXiv:2106.0966 2021.
[27] Jia Liu, Tianrui Li, Shenggong Ji, Peng Xie, Shengdong Du, Fei Teng, and Junbo Zhang. Urban flow pattern mining based on multi-source heterogeneous data fusion and knowledge graph embedding. IEEE Transactions on Knowledge and Data Engineering, 2021.
[28] Shiyuan Zhang, Tong Li, Shuodi Hui, Guangyu Li. Yanping Liang, Li Yu,, Depeng Jin, and Yong Li. Deep transfer learning for city-scale cellular traffic generation through urban knowledge graph. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4842-4851, 2023.
[29] Eunjoon Cho, Seth A Myers, and Jure Leskovee. Friendship and mobility: user movement in location- based social networks. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, pages
2011.
[30] Inderject Mani, Christy Doran, Dave Harris, Janet Hitzeman, Rob Quimby, Justin Richer, Ben Wellner, Scott Mardis, and Seamus Clancy. Spatialml: annotation scheme, resources, and evaluation. I Language Resources and Evaluation, pages 263-280. 2010.
[31] Rohin Manvi, Samar Khanna, Gengehen Mai, Marshall Burke, David Lobell, and Stefano Ermon, Geollm: Extracting geospatial knowledge from large language models. arXiv preprint arXiv:2310.06213, 2023.
[32] Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, et al. Zero-shot information extraction via chatting with chatgpt. arXiv preprint arXiv:2302. 2023.
[33] Aohan Zeng. Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for 1lms. arXiv preprint arXiv:2310. 12823, 2023.
[34] Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. Fireact: Toward language agent fine-tuning. arXiv preprint 1rX0v:2310.05915, 2023.
[35] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of thtohhh prompting elicits reasoning in large language models, pages 24824 24837,24877558 2022.
12

---

[36] Tianle Cai, Xuezhi Wang. Tengyu Ma. Xinyun Chen, and Denny Zhou. Large language models as tool makers, arXiv preprint drXiv:2305.17126, 2023.
[37] Ruixin Hong. Hongming Zhang. Hong Zhao, Dong Yu, and Changshui Zhang. Faithful question answering with monte-carlo planning. arXiv preprint 1rXiv:2305.0255: 2023.
[38] Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. arXiv preprint 1r00:223,,,01998 2023.
[39] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303. 17651, 2023.
[40] Keming Lu. Xiaoman Pan, Kaiqiang Song. Hongming Zhang, Dong Yu. and Jianshu Chen. Pivoine: Instruction tuning for open- world information extraction. arXiv preprint arXiv:2305. 20.::2305.4498. 2023.
[41] Hongbin Ye, Ningyu Zhang, Hui Chen, and Huajun Chen. Generative knowledge graph construction: A review. arXiv preprint wwxi:22200.12111 2022.
[42] Yew Ken Chia, Lidong Bing, Soujanya Poria, and Luo Si. Relationprompt: Leveraging prompts to generate synthetic data for zero-shot relation triplet extraction. arXiv preprint arXiv:2203.09101, 2022.
[43] Liang Yao, Chengsheng Mao, and Yuan Luo. Kg-bert: Bert for knowledge graph completion. arXiv preprint
2019.
[44] Apoorv Saxena, Adrian Kochsiek, and Rainer Gemulla. Sequence-to-sequene knowledge graph comple- tion and question answering. arXiv preprint arXiv:2203. 10321, 2022.
[45] Cunxiang Wang, Sirui Cheng, Zhikun Xu, Bowen Ding, Yidong Wang, and Yue Zhang. Evaluating open question answering evaluation. arXiv preprint arXiv:2305. 12421, 2023.
[46] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vvccna. Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
[47] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-folloing Ilama model. 2023.
[48] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint ar 130002332000023. 068555 2023.
[49] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
[50] AI@ Meta. Llama 3 model card. 2024.
[51] OpenAL. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[52] Lianmin Zheng, Wei-Lin Chiang. Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li. Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arx: 05685, 2023.
[53] Siqi Lai,, Zhao Xu, Weijia Zhang, Hao Liu, and Hui Xiong. Large language models traffic signal control agents: Capacity and opportunity. arXiv preprint arXiv:2312. 16044, 2023.
[54]
Signifiant-graviiaaaattttoop An experimental open-source attempt make gpt-4 fully autonomous. 2023.
[55] Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bemard Ghanem. Camel: Communicative agents for" mind" exploration of large scale language model society. arXiv preprint 2oXiv:2303.17766 2023,
[56] Zheng Gong and Ying Sun. Graph reasoning enhanced language models for text-to-sql. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2447-2451, 2024.
[57] Guozheng Li, Peng Wang, and Wenjun Ke. Revisiting large language models as zero-shot relation extractors. arXiv preprint arXiv:2310.05022 2023.
13

---

[58] Chenhan Yuan, Qianqian Xie, and Sophia Ananiadou. Zero-shot temporal relation extraction with chatgpt. arXiv preprint (rXiv:230.005544 2023.
[59] Yueling Zeng and Li-C Wang. Domain knowledge graph construction via a simple checker. arXiv preprint vrXXv:2310.04949, 2023.
[60] Chengbiao Yang and Guilin Qi. An urban traffic knowledge graph-driven spatial-temporal graph convolu- tional network for traffic flow prediction. In Proceedings of the 11th International Joint Conference on Knowledge Graphs, pages 110-114, 114, 2022.
[61] Jiyuan Tan, Qianqian Qiu. Weiwei Guo, and Tingshuai Li. Research the construction of knowledge graph and knowledge reasoning model the field of urban traffic. Sustainability. page 3191, 2021.
[62] Ling Zhao, Hanhan Deng, Linyao Qiu, Sumin Li, Zhixiang Hou, Hai Sun, and Yun Chen. Urban multi- source spatio-temporal data analysis aware knowledge graph embedding. Symmetry, page 199, 2020.
[63] Zhilun Zhou, Yu Liu, Jingtao Ding, Depeng Jin, and Yong Li. Hierarchical knowledge graph learning enabled socioeconomic indicator prediction in location-based social network. Proceedings of the ACM Web Conference, pages 122-132, 2023.
[64] Ruiyi Yang, Flora Salim, Hao Xue. Sstkg: Simple spatio-temporal knowledge graph intepretable and versatile dynamic information embedding. arXiv preprint arXiv:2402.12132 2024.
[65] Jindong Han, Hao Liu, Hengshu Zhu, and Hui Xiong. Kill two birds with one stone: multi-view multi-adversarial learning approach for joint air quality and weather prediction. IEEE Transactions on Knowledge and Data Engineering, 35(1 (5(11):11515 11528, 2023.
[66] Qinyuan Cheng, Tianxiang Sun, Wenwei Zhang, Siyin Wang, Xiangyang Liu, Mozhi Zhang, Junliang He. Mianqiu Huang. Zhangyue Yin, Kai Chen, et al. Evaluating hallucinations in chinese large language models. arXiv preprint ar 2rXiv:2310003368. 2023.
[67] Jinlan Fu, See- Kiong Ng. Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as desire. arXiv preprint arXiv:3302.44166 2023.
[68] Jemoldd H Zar. Spearman rank correlation. Encyclopedia of biostatistics, 2005.
14

---

Table The detailed statistic of RTE datasets. We report the maximum length, minimum length, and average length of urban text in the RTE dataset.
Dataset
Max Length
Min Length
Avg Length
Records
NYC-Instruct
1,747
68
437
232
CHI-Instruct
1,120
25
408
122
NYC
2,708
51
433
2,089
CHI
1,883
32
445
1,102
NYC-Large
4,598
20
1,179
40,480
CHI-Large
4,597
36
825
28,868
Figure 7: The geometry range visualization of the head entity and tail entity of four KGC datasets. The horizontal and vertical coordinates are longitude and latitude, respectively. The blue and red polygons stand for entities with the polygon geometry, the purple line string stands for the entities with linestring geometry and the green point is for the coordinate entities.
A UrbanKGC Construction
This section presents detailed UrbanKGC dataset statistic information for relational triplet extraction (RTE) and knowledge graph comple- tion (KGC) tasks. Since two sequential tasks (i.c.. RTE and KGC) of UrbanKGC are within an owwn-ppeeieeeeooooe world setting (i.c., no predefined ontol- ogy) 40 4T. Therefore, for the RTE task, every data record is an urban text without the triplet label. For the KGC task, every data record is a quadruple (i.e., head entity name, head entity geometry, tail entity name, tail entity geometry) without the geospatial relation label.
Y
X DCY
X
X POY
X BQY
X INY
Figure Given two geo-entities X and Y, the illustrative visualization of five types of RCC rela- tionships. this work, we consider entities with coordinate geometry to circles with very small radii.
RTE Dataset. To facilitate the understanding of the constructed RTE dataset, we summarize the distribution of urban textual corpus in the six RTE datasets. As shown in Table 5 the entity
distribution and textual statistics of the instruct dataset and test data are similar. In addition, the record in dataset (i.e., NYC-Instruct and CHI-Instruct) used for instruction tuning is not overlapping with that in test dataset (i.e., NYC and CHI) and the real-world application dataset (i.e., NYC-Large and CHI-Large). This can avoid potential data leakage issues.
KGC Dataset. As for the KGC dataset, we provide illustrative visualization of the five RCC relationships 30 Figure for better understanding. Specifically, the disconnected (DC), externaly connected (EC), partially overlapping (PO), equal (EQ), tangential and non-tangential proper parts
15

---

Table 6: The detailed geospatial tool name, description, interface input, and output. Each tool interface implemented by python and it is self-programmed by GPT-4.
Tool name
Tool description
Input
Output
Goobash
Geohash cacoding
Goometry
Goohash code (8-bit)
Distance
Calculate the distance between geo entities.
Goometry Geometry
Distance value (km)
Point2Polygon
Identify if point belongs to a polygon
Point geometry, polygon geometry
True/Fakee
Point4Linestrigg
Identily if point intersects linestring Identify if linestring belongs to a polygon Identify if linestring intersects poly polygon gun Ideatity if polygon belongs to a polygon Identify if polygon intersects polygon
Point geometry, linestring geometry
True/False
Linestring geomeiry, polygon geometry Linestring geometry. polygon geometry Polygon geometry. polygon geometry Polygon geometry, polygon goometry
True/Take
LinestriggPPolyge
True/False
Polygon2Polygod
Trae/False
JlyggnnPPllygon
Trae/Palke
(IN) together depict the basic geospatial relationship between urban entities. Moreover, to facilitate the understanding of the constructed KGC dataset, we visualize the geometry range of the head entity and the tail entity in four small KGC datasets. Due the large amount of data and the display overlapping between entities and entities during visualization, we will not show the visualization results of NYC-Large and CHI-Large dataset, but the pattern is similar. As can be seen in Figure 7 the entity distribution in the instruct dataset (i.e., NYC-Instruct and CHI-Instruct) and test dataset (i.e., NYC and CHI) are similar. The pattern in NYC-Large and CHI-Large is similar. Due the KGC task follows a zero-shot setting, we are unable to provide accurate distribution proportions of the 5 RCC relationships in the four datasets. However, it is intuitive be seen that all four datasets contain the five RCC relationships shown in Figure 6 which provides a guarantee for the practical significance of KGC task in this work.
B Instruction Template
B.1
Instruction Template in UrbanKGent
This section presents the detailed instruction template of the proposed UrbanKGent framework. Specifically, Figure([j--b) provides the detailed instruction template of knowledgeable instruction generation module and tool-based trajectory augmentation. The iterative trajectory self-refinement module is achieved by trajectory verifier and updater in Figure
B.2 Instruction Template in Baselines
We provide the detailed instruction template of all the baseline models in this work.
LLMs-based Zero-shot Reasoning Methods. We only provide task descriptions for the zero-shot reasoning method. The detailed instruction template is shown in Figure
LLMs-based In-context Learning Methods. We first construct several few-shot demonstrations
via the chain-of-though prompting techniques [35), which is popular for automatic demonstration
generation. The detailed instruction template shown in Figure g Then, we add these demonstrations
before the test question, and the detailed instruction template is shown in Figure
Vanilla Fine-tuning Methods. We only provide task descriptions for vanilla fine fine-tuning methods without demonstrations. The detailed instruction template is shown in Figure 9
UrbanK Gent Inference Methods. These baseline models follow the same inference pipeline shown in Figure g
C Geospatial Toolkit
This work constructs a geospatial reasoning toolkit (eight interfaces in total Table by asking GPT- 4 for relf-programiin. We obtained geospatial toolkit for point, linestring, and polygon geometry objects, supporting distance calculation and Geohash encoding for two geographical objects, and can also determine basic spatial relationships between two geometric objects. such as containment and intersection. For each interface, we provide its name and function description (e.g., geohash encoding in Table and Figure 8b) into the instruction template.
16

---

Darch Aeteller Tarte
ADd thi
fun thi
dnserien prtr brady vee
Thocmito Blay dankk sulelarr
listtl intation
Soctact
tenter mostte ape famee
(epere: rileio tys
ber (oaae ridate can
Puor bid mode
you snd
Rediuled Eay'
<rome chcotaar
Tell
phote
oour
neturey
in
feam
Figure 8: The reasoning process of UrbanKGent- and the detailed instruction template for UrbanKent inference pipeline. The content in the gray box is the instruction, and the content in the green box is the agent's 's response.
Sdancaa Scpentle ottete
Mus organu syrus Aerty adrtce
Antgmry wes phote sngir pha
Nocfh Vin araay reggiie
carr
the
Bale's
taft
Tux
Figure 9: The instruction template of zero-shot reasoning, In-context learning and vanilla fine-tuning baselines.
17

---

Table 7: Illustrative RTE evaluation example when we utilize human evaluation and GPT evaluation method. We calculate the accuracy by counting the proportion of true triplets. The label for GPT evaluation method is invisible.
Type
Urhan text
Results
Label
Number of the triplet
Number the Fulse triplet
Haman Evaluation
<Columbia University, locato-in, New York City
Columbia University private Ivy league research university New York City.
<Columbia University. locate-in, New York City
GPT Evaluation
0
D Evaluation
This section presents the detailed evaluation process and examples help the reader have better understanding: (1) Human Evaluation. We employ human annotators to evaluate the results on 200 random samples. For the relational triplet extraction task, we first manually annotate the triplet label for each sample. Then, we manually evaluate the correctness of each triplet [40] based on annotation and calculate the accuracy value. For the knowledge graph completion task, we follow [[]] to manually label the response as correct or wrong, and calculate the accuracy. (2) GPT Evaluation.
Recently, many studies [52] 45 adopt LLM-based evaluation for open-domain tasks and empirically demonstrate that GPT-4's evaluation and human evaluation can be consistent [66]. In this work, we also use GPT-4 to evaluate the model performance on the full data to escape intensive labor. Specifically, given an UrbanKGC instruction and results, we prompt GPT-4 to return the confidence score and the justification (i.e., True/False), which will be further used to calculate the accuracy.
Moreover, we provide a comprehensive analysis to demonstrate why GPT and Human evaluations are highly aligned.
D.1 Human Evaluation Process
For the relational triplet extraction (RTE) task, we provide an evaluation example in Table 7 Given the urban text sentence "Columbia University is a private Ivy league research university in New York City.", the human annotators are required to first label the triplet contained in this urban text, described as triplet <Columbia University, locate-in, New York Cityz. Based on the label, then, they are instructed to evaluate how many true triplets and false triplets in the results from the models. Finally, they will fill out the evaluation form (i.e., the number of the true triplets and the number of the false triplets). We will calculate the accuracy of results based on these annotated forms.
For the knowledge graph completion (KGC) task, given the head entity <Columbia University, Point (40.8075, 73.9626)> and the tail entity <Empire State Building, Point (40. 74047 73.9857)>, the human annotators are first required complete their missing geospatial relationship from the five relation candidate (i.e., DC, EC, PO, EQ and IN). Specifically, the annotation could be achieved by manually visualizing the location of two entities given on the map and following the RCC relation rule in Figure oo to determine their geographical relationships. Finally, they will fill out the evaluation results (i.e., True/False). We will calculate the accuracy based on these evaluation results.
D.2 GPT Evaluation Process
For the relational triplet extraction (RTE) task, we provide an evaluation example in Table 7 Given the urban text sentence "Columbia University is a private Ivy league research university in New York City.", and the model extraction results (i.e., <Columbia University, locate-in, New York City>), we directly prompt GPT-4 to fill out the evaluation form (i.e., the number the true triplets and the number of the false triplets). Then, the accuracy could be obtained based on these self-evaluated results.
For the knowledge graph completion (KGC) task, given the head entity <Columbia University, Point (40.8075, 73. (3.9626>> and the tail entity <Empire State Building, Point (40. 24077484, 73.9857)> and completed geospatial relationship, we directly prompt GPT-4 the justify if the results are correct. Specifically, we will first explicitly call eight external geospatial tools in Table and combine all these eight calculation results into the prompt. Then feed these prompts into GPT-4 to help
18

---

RITSCCE
141
KOO MX0
19
1.5
Figure 10: The Spearman correlation between the GPT evaluation and human's evaluation under five
different LLM backbones (i.e., Llama-2-7B, Llama-2-13B, Llama-2-70B, GPT-3.5 and GPT-4).
B B
VA Libwak
V4 Ubra
KCONYC
K0009
Figure 11: The Spearman correlation between the GPT evaluation and human's 's evaluation under four LLM paradigms (i.e., Zero-shot learning (ZSL), In-context learning (ICL), Vanilla fine-tuning (VFL) and UrbanK Gent inference).
make a comparable accurate evaluation. Finally, we obtain the evaluation results (i.c., True/False) for every record. We will calculate the accuracy based on these evaluation results.
However, such an evaluation process carries risks as we cannot guarantee the infallibility GPT-4. To ensure the validity of the evaluation, we have conducted extensive experiments to demonstrate that the GPT evaluation method aligns well with human evaluation.
D.3 Evaluation Consistency
We conduct experiments to assess the consistency between GPT-4's evaluation results and human evaluation results. In this work, we follow GPTScore [67 to use Spearman coefficient [68] to investigate the correlation between GPT evaluation and human evaluation. Based on the evaluation results in Tabel B can conduct correlation analysis separately by the types of model and paradigm.
Comparison of LLM Backbones. As shown in Figure I0 we observe that the Spearman correlation coefficients on all LLM backbones are greater than 0.8, which means the human evaluation and GPT evaluation on these five LLM backbones highly correlated.
Comparison of LLM Paradigm. Moreover, we also conduct correlation under different LLM paradigms for UrbanKGC tasks. As be seen in Figure I1] all four types of paradigm have a higher Spearman value above 0.85. Therefore, deriving the GPT evaluation method is also applicable to different LLM paradigms when finishing UrbanKGC tasks its evaluation aligns well with that of the human being.
Analysis of Evaluation Repeat. Nevertheless, the above correlation analysis is based results in Table 3, which are obtained after only single evaluation. To deeply understand the evaluation mechanism, We repeatedly instruct GPT-4 to generate 1, 5, and 10 evaluation results understand the variance of the evaluation method. To save the cost, we perform the above analysis using UrbanKGent-78 only the NYC dataset. The results shown in Table R although the coefficient value will decrease as the number of repeated experiments increases. However its value has always remained within a high correlation range (roughly 0.85).
E In-Depth Analysis
In this section, we aim to perform more comprehensive analysis of our proposed UrbanKGent agent and hope to answer the following research questions. (1) RQ1: How does the constructed
19

---

Table 8: The average Spearman correlation value between human evaluations and GPT-4 evaluations. "Repeat X times" refers to instructing GPT-4 to generate judgments X times, and adopting the answer that appears most frequently (e.g.. True/False for the KGC task and Number of the true triplet for the RTE task) as the final decision.
Task
Repeat Once
Repeat 3 times
Repeat 5 times
Repeat 10 times
RTE
98.56% 94.86%
87.31
86.42%
85.27%
KGC
91.23%
90.09%
91.24
Table 9: The experimental results of relational triplet extraction (RTE) and knowledge graph comple-
tion (KGC) the NYC-Large dataset and CHI-Large dataset. To save the cost, We choose the best
approaches in zero-shot-learning (ZSL), In-context-learning (ICL), Vanilla Fine-tuning (VFT), and
UrbanKGent Inference setting.
NYC-Large
CHI-Large
Models
GPT(ace/confidence)
Human(acc) RTE
GPT(acc/confideecc)
Human(ace) RTE
RTE
KGC
KGC
RTE
KGC
KGC
7SL.
0.36/4.01
0.45/3.95
0.42
0.31
0.38/4.27
0.33/3.67
0.44
0.36
ICL
0.39/4.36
0.48/4.15
0.48
0.39
0.38/4.22
0.35/3.53
0.40
0.35
VFT
0.37/4.05
0.46/3.98
0.45
0.35
0.33/4.25
0.28/3.67
0.38
0.34
UrbanKGent Inference
0.43/4.13
0.51/3.87
0.51
0.43
0.47/4.33
0.43/3.63
0.49
0.42
Urban 7
0.44/4.27
0.50/4.07
0.53
0.44
0.48/4.25
0,42/3.85
0.55
46
UrbanKGent-8B
0.43/4.07
0.51/4.16
0.52
0,44
0.49/4.21
0.43/3.77
0.53
44
UrbanKGent- 13B
0,46/4.56
0,52/3.67
0.55
0.46
0.55/4.29
0.49/3.24
0.58
0.49
UrbanKGent perform compared with existing paradigms on larger real-world dataset? (2) RQ2: How do different components (e.g., the knowledgeable instruction generation) affect the performance? (3) RQ3: How the complexity and efficiency of proposed UrbanKGent framework? (4) RQ4: How about the UrbanKGent inference trajectories when completing UrbanKGC tasks? (5) RQ5: How can the constructed UrbanKGent provide application service to real-world scenarios?
E.1 RQ1: Evaluation on Larger Dataset
As shown in Table 9 we derive UrbanKGent- 13B for urban knowledge graph construction using constructed large-scale dataset in NYC and CHI. Specifically, we directly use the text record in NYC-Large and CHI-Large for the relational triplet extraction task. Then, we randomly sample the head-tail entity pairs (both of head and tail entities contain geometry information) from these triplets for knowledge graph completion. Nevertheless, iterating all head-tail entity pairs is time-consuming task, so we just construct a KGC dataset consistent with the sacle of the RTE dataset. By performing RTE and KGC tasks, obtain two large-scale UrbanKGs shown in Table Compared with existing dataset and benchmark UUKG [8, we can clearly observe that our agent can only use one- fifth data construct the UrbanKG with the same scale entities and triplets, but extend the relationship types to a thousand times. It is worth noting that all construction process completed by a LLM agent without any mannul effort. We think it is the core of this work.
Moreover, we also report the performance of the RTE and KGC in NYC-Large dataset. To save the cost of GPT self-evaluation, we only choose the best approaches in ZSL, ICL, VFT and UrbanKGent Inference. The experimental results are shown Table As can be seen, the fine-tuned Ur agent, whether 7B or 13B version, could achieve state-of-art performance UrbanKGC tasks.
E.2
RQ2: Ablation Studies
We conduct an in-depth analysis of the proposed instruction generation and tugm-nngee iterative trajectory refinement module on the NYC dataset. Specifically, for the RTE and KGC task, we validate the effectiveness of each block by comparing the following variants: (1) Urban G removes knowledgeable instruction template RTE and KGC task; (2) UeGent" removes multi-view design in RTE task; (3) UrbanKGent-7BB removes external geospatial tool invocation block; (4)
20

---

Table 10: Effect of different blocks.
GPT (acc/confidence)
Human (ace)
Models
RTE
KGC
RTE
KGC
UrbanKGent-72
0.38/4.17
0.42/3.98
0.37
0.34
UrbanKGent-7BB
0.34/4.06
0.45/4.02
0.34
0.39
UrbanKGent-7BB
0,45/4.32
0.40/3.97
0,45
0.23
UrbanKGent-7Bt
0.44/4.10
0.47/3.85
0.46
0.43
Table 11: Comparison among LLM-based UrbanKGC methods in four ways.
Extra Knowledge
Require Fine-tuning
Tool Invokation
Self Refinement
Method
ZSL
ICL
XX
X
VFT
V
X
UrbanKGent Inference
GrnanKGent
V
UrbanKGent-7B" removes iterative trajectory self-refinement. We summarize the results in Table 10 and obtain the following observations.
First, knowledgeable instruction generation contributes to the overall performance of both RTE and KGC tasks. We observe performance degradation by removing the knowledgeable instruction template. Second, the multi-view instruction design provides the most performance gain, which matches our intuition that the UrbanKG text contains heterogeneous relationships that be effec- tively extracted by multi-view prompting design. Third, the tool invocation is very important for the KGC task, as we observe significant performance degradation after removing the tool invocation. In addition, the iterative trajectory self-refinement brings consistent performance gain for both the RTE and KGC tasks.
E.3
RQ3: Complexity and Effiency Analysis
We make a comparison with the four paradigms to demonstrate the advantages of the constructed agent, which is shown in Table Compared with Zero-shot reasoning (ZSL.), In-context Learning (ICL), Vanilla Fine-tuning (VFT), and UrbanKGent Inference, UrbanKGent can incorporate extra urban knowledge, invoke external tools and iteratively self-refine to help better complete UrbanKGC tasks.
Moreover, we also provide comprehensive efficiency analysis show the latency and cost of different models when completing UrbanKGC tasks. Specifically, we report the total inference time and cos of each method completing with 1,000 RTE and KGC tasks. For the cost of GPT-4 service, we first count the number of prompt token and completion token spent on 1,000 RTE and KGC tasks, and then calculate the cost based on billing standards. As for the cost of UrbanKGent- 13B, we first count the GPU running time spent on 1,000 RTE and KGC tasks, and then calculate the cost based on the A800 charging standard.
In addition, as reported in table 12 we also provide detailed inference latency of UrbanKGent family when deriving them for constructing different-scale of UrbanKGs.
E.4
RQ4:
Case Study
As shown in Figure g we present the detailed reasoning process of constructed UrbanKGent- when finishing urban relational triplet extraction and knowledge graph completion task. Since the
We subscribe to the NVIDIA A800 computing resources and GPT service from HKUST(GZZ, Following the standard price instruction, could calculate cost of GPT-based baselines Llama based baselines.
21

---

Table 12: The inference latency comparison of UrbanKGC using UrbanKGent family. We use two middle-size dataset (i.e., NYC and CHI) and two large-scale dataset (i.e., LyceL and CHI-Large) for UrbanKG construction.
Latency (minutes)
Datasct
Data Volume
UrhanKGGen-77B
UrhanKGent-8B
VihanKGGen--I31
NYC
1.19
1,58
3.19
2,089
CHI
0.62
1.13
1.68
1,102
NYC-Large
23.07
30.76
61.93
40,480
CHI-Large
16.55
21.93
44.47
28,868
0.57
0.76
1,53
Per 1,000 records
Table 13: The statistic of entity and relation ontology of constructed UrbanKGs on NYC-Large and CHI-Large datset.
# Ccarse-grainee
Fine-grained Entity Ontology
Coarse-grained Relation Ontology
Fine-grained Relation Ontology
UrbanKG Dataset
# Entity
Triplet
Entity Ontology
NYC-Large CHI-Large
6,281
2,138 1,336
228,928 95,813
905,442 563,290
4
2,559
iterative self-refinement process contains excessive text, we display the reasoning process of only one iteration.
E.5
RQ5: Agent Application
We have released the UrbanKGent family consisting of 7B, 8B and 13B version in the Huggingface. The opensourced UrbanKGent family offer urban knowledge graph construction service for the researcher in this field. We provide application example in New York and Chicago. Specifically, following the UrbanKGent Inference framework, sequentially derive UrbanKGent-133 for relational triplet extraction and knowledge graph completion. The obtained initial UrbanK encodes diverse urban spatial, temporal and functional knowledge. Then, we propose use two-stage triplet filtering and relation merging operation to further improve the quality of constructed UrbanKGs.
In the first stage, low- freeuenccycy relations (occur 5 times less) merged into high-frequenee relations based on relation similarity threshold. The remaining low-frequency triples, whose similarity with any high-frequent freeuerrynncy relation is below the threshold will be filtered out. In the stage two, first perform relation clustering based on the embedding of relations. Then, Within each cluster, prompt LLM to identify semantically similar relations that can be merged into single relation
category, resulting the final set of merged relations.
As shown in Table 13 both urban entity and relation can be pre-categorized into 4 coarse-grained ontologies: spatial, temporal, functional, and others. The entity percentage spatial, temporal, functional and others in NYC-Large is (68.34%, 17.66%, 10.39%, 3.61 %), and in CHI-Large is (63.29%, 16.68% 12.09%. 7.94%). The relation percentage of spatial, temporal, functional and others in NYC-Large and CHI-Large are (57.36%, 16.38% 20.89%, 5.37%) and (60.57%, 15. 11%, 21.67% and 2.65 (),, respectively. After multi-view entity recognition and relation extraction (shown in Figure 4 () in our paper), the fine-grained entity ontologies (6,281 and 2,559 entity types of NYC-Large and CHI-Large JrranKGGs respectively) and fine-grand ane-grained relation ontologies (2, and 1,366 relation types of NYC-Large and CHI-Large UrbanKGs,, respectively) are obtained.
F Limitation and Future Work
This work has limitation on the further application demonstration of construction UrbanKGs, although proposed UrbanKGent family could construct UrbanKG with hundreds relationship using only one-fifth of data. In addition, the evaluation method in this work is cost-intensive although GPT evaluation and Human evaluation has been experimentally demonstrated to consistent. Despite the above limitations, we hope the opensource UrbanKGC agent can foster more extensive UrbanKG research and broad smart city application. In the future, we will derive extra image- madge-m data to further enrich Ur.
22

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
In the end of abstract and introduction, the main claims are made.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected generalize other setrings.
It is fine include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not diseussed in the paper.
The authors are encouraged create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution low or images are taken low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess..
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and
complete (and correct) proof?
Answer: [NA]
23

---

Justification:
This paper focus on domain application, no theory assumptions proofs provided in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Researcher could follow the provided code reproduce experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided not.
If the contribution a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g.. in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer:[Yee Yes]
Justification:
The code and data are in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips.. cc/
public/gu gu Codeeubmm ss Poooooooiii plyyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips co /tappp/so//puliiicciiiieee/////s
ionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
submission time, to preserve anonymity, the authors should release anonymized ver- sions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All training details could be found in the code of the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Necessary statistical indicator of the results are provided in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
25

---

It is OK to report 1- I-sigma enror bars, but one should state it. The authors should preferably report 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the experimental setting section 5.1 detailed computing resources explana- tion is provided.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't t make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the http:///erriiii
NeurlPS Code of Ethics http:///arrrrs...cooooooocccttoa
Answer: [Yes]
Justification:
This paper follows the NeurlPS Code Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethies.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: In the end of introduction and agent application section 5.3 we discuss how can our proposed LLM agent to be deployed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses addition to attacks, mecha- nisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The data and models used in this paper are opensourced, there is no critical issue model and data.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The researcher follow CC-BY 4.0 license to use the data and code in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include a URL.
The name of the license (c.g.. CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, [apper swi thcode com/ hat//aatssssss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification:
We provide illustrative details for the new dataset construction in Section 2
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their sub- missions structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
The human evaluation details could be found in Appendix D
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
28


Reviews:
[{"review_id": "m44rQoMfYU", "reviewer": "Reviewer_72X8", "summary": "This paper proposes a unified large-scale language model agent framework called UrbanKGent, specifically for the Construction of Urban Knowledge Graph Construction (UrbanKGC). UrbanKGent utilizes instruction generation of heterogeneous and geospatial information, as well as an iterative trajectory optimization module based on GPT-4, to further enhance the ability to extract critical knowledge from multi-source city data and effectively reduce the significant manual labor of traditional methods. By fine-tuning the hybrid instructions on the Llama 2 and Llama 3 series models, the researchers developed the UrbanKGC agent, including the UrbanKGent-7/8/13B version. Experiments on two real-world datasets show that the UrbanKGent family not only significantly outperforms 31 benchmark models on the UrbanKGC task, but is also more than 20 times more cost-effective than GPT-4, while being able to build a knowledge map of cities with hundreds of times richer relationships with less data.", "questions": "NA", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. UrbanKGent proposed in this paper provides a unified solution for building urban knowledge graphs, which automates the process of extracting key information from multi-source urban data and reduces the need for human intervention.\n\n2. The authors propose instruction generation methods of \"heterogeneity awareness\" and \"geospatial information fusion\", which can better capture the characteristics of urban knowledge graph construction tasks and make up for the shortcomings of ordinary LLM in understanding complex heterogeneous relationships and geospatial computing capabilities.\n\n3. Experimental results on two real-world data sets show that UrbanKGent not only significantly outperformed 31 benchmarks on the UrbanKGC task but also outperformed the state-of-the-art LLM GPT-4 by more than 10% at about 20 times lower cost, which has great potential for efficiency and economy in practical applications.", "weaknesses": "1. As the size of the dataset grows, the algorithmic efficiency and scalability of UrbanKGent may become an issue. Although not mentioned in the paper, in practical applications, the processing of large-scale data sets may require more computational resources and time. Although the paper notes limitations, the specifics may require more elaboration. For example, under what conditions may the model perform poorly and how these limitations affect the quality and reliability of the final knowledge graph?\n\n2. UrbanKGent may face privacy and fairness challenges, especially when handling personal or sensitive information. I wonder what the author thinks about privacy and fairness. Although these issues are not discussed in detail in the paper, in actual deployment, if not properly addressed, they can lead to violations of personal privacy or unfairly affect specific groups.\n\n3. The UrbanKGent framework is mainly aimed at urban knowledge graph construction tasks. I wonder if the author has considered the applicability of other types of knowledge graph construction tasks. Can the UrbanKGent framework proposed in this paper be further extended to the knowledge graph construction in other fields?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> [W1] As the size of the dataset grows, the algorithmic efficiency and scalability of UrbanKGent may become an issue. Although not mentioned in the paper, in practical applications, the processing of large-scale data sets may require more computational resources and time. Although the paper notes limitations, the specifics may require more elaboration. For example, under what conditions may the model perform poorly and how these limitations affect the quality and reliability of the final knowledge graph?\n\nThanks for the reviewer's insightful question, which helps us improve the quality of our paper. We agree with the reviewer that efficiency is important in practical agent deployment. As suggested, we provide a more detailed efficiency analysis of UrbanKGent in the following table, and we also use LLM acceleration techniques VLLM [1] to improve the scalability of UrbanKGent. As shown in Table 1, by applying VLLM techniques to accelerate our current framework, UrbanKGent-13B can automatically construct an UrbanKG with approximately one million triples (905,442 extracted from the NYC-Large dataset) in about 62 minutes. We also report the average inference time of the UrbanKGent family. As can be seen in Table 1, For every 1,000 data records processed, UrbanKGent takes 0.57, 0.76, and 1.53 minutes on 7B, 8B, and 13B versions, respectively.\n\nTable 1: Comparison of UrbanKGC inference latency (minutes) using the UrbanKGent family before and after using VLLM acceleration. We use two middle-size datasets (i.e., NYC and CHI) and two large-scale datasets (i.e., NYC-Large and CHI-Large) for UrbanKG construction. The accelerated inference latency is **bolded.**\n\nDataset|Latency (minutes)|||Data Volume\n-|-|-|-|-\n|| UrbanKGent-7B     | UrbanKGent-8B    |UrbanKGent-13B|\nNYC|6.83/**1.19**|7.62/**1.58**|16.48/**3.19**|2,089\nCHI|3.60/**0.62**|4.02/**1.13**|8.69/**1.68**|1,102\nNYC-Large|132.33/**23.07**|147.75/**30.76**|319.38/**61.93**|40,480\nCHI-Large|94.39/**16.55**|105.36/**21.93**|227.76/**44.47**|28,868\n-|3.27/**0.57**|3.65/**0.76**|7.89/**1.53**|Per 1,000 records\n\nCurrently, the maximum dataset scale we used is about forty thousand. We are working on processing larger-scale data and analyzing the potential performance degradation and KG quality issues. Thanks to the reviewer for introducing these interesting research questions, and we will discuss them in the Section \"Limitation and Future Work\" in our paper.\n\n> [W2] UrbanKGent may face privacy and fairness challenges, especially when handling personal or sensitive information. I wonder what the author thinks about privacy and fairness. Although these issues are not discussed in detail in the paper, in actual deployment, if not properly addressed, they can lead to violations of personal privacy or unfairly affect specific groups.\n\nThanks for the reviewer's insightful question. Currently, we collect raw urban data from various public data providers (e.g., OSM, Wikipedia) to construct UrbanKGent. Although sensitive information is not a critical issue in these open-source data, we agree with the reviewer's concern regarding potential data privacy and fairness issues once UrbanKGent is deployed. To address this, we can further perform safety alignment [1] in UrbanKGent to control its outputs. In addition, we can subscribe to external LLM services to filter the malicious information of UrbanKGent's output. The above two strategies have been widely used in many online LLM reasoning services (e.g., ChatGPT, Ernie Bot, and Tongyi Qianwen), and we believe they would be the practical solution. We will further discuss this in the \"Limitations and Future Work\" section of our paper to provide more insights.\n\n**Reference**:\n\n[1] Ji, Jiaming, et al. \"Beavertails: Towards improved safety alignment of llm via a human-preference dataset.\" *Advances in Neural Information Processing Systems* 36 (2024).\n\n> [W3] The UrbanKGent framework is mainly aimed at urban knowledge graph construction tasks. I wonder if the author has considered the applicability of other types of knowledge graph construction tasks. Can the UrbanKGent framework proposed in this paper be further extended to the knowledge graph construction in other fields?\n\nThanks for your insightful comment and question! Yes, although the proposed framework is designated to the urban domain, the construction pipeline can be easily extended to the construction of knowledge graphs for other domains. One of the most straightforward ways is to modify the instruction template to adapt this framework to other domains. For example, we can replace the urban knowledge currently encoded in the RTE instruction template with other domain knowledge, to adapt our framework for the knowledge graph construction in other fields.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I9VBdZfVSC", "reviewer": "Reviewer_GYCX", "summary": "The paper presents UrbanKGent, a unified large language model agent framework for urban knowledge graph construction. The framework consists of knowledgeable instruction generation, tool-augmented iterative trajectory refinement, and hybrid instruction fine-tuning. The authors evaluate the framework on two real-world datasets using both human and GPT-4 self-evaluation. The experimental results show that UrbanKGent outperforms 31 baselines in urban knowledge graph construction tasks and achieves state-of-the-art performance.", "questions": "1. What is the ontology of urban KGs?\n2. What is the motivation for selecting geospatial relations for KGC tasks and using LLMs for this? Can we predict other relations using the same framework? Can we directly use the tool to get relations if geospatial information is given?\n3. Please provide more analysis of the iterative trajectory self-refinement module.\n4. Settings of the ablation studies.\n5. Cost for fine-tuning data construction.\n6. Can we compare UrbanKGent with existing UrbanKG construction methods?", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper proposes a unified framework for urban knowledge graph construction, addressing the challenges of heterogeneous relationship understanding and geospatial computing.\n\n2. The knowledgeable instruction generation module and tool-augmented iterative trajectory refinement module are innovative and practical methodologies.\n\n3. The experimental evaluation is comprehensive, including both human evaluation and GPT-4 self-evaluation.\n\n4. The results show that UrbanKGent outperforms 31 baselines in urban knowledge graph construction tasks and achieves state-of-the-art performance.", "weaknesses": "1. This paper focuses on urban knowledge graph construction tasks. However, the details of the urban knowledge graph are not clearly defined. For example, the ontology of urban KGs, entities types, and relations types are not discussed. This limits the understanding of the proposed framework for readers who are not familiar with urban knowledge graphs.\n\n2. The KGC tasks only focus on the geospatial relations. Is there any specific reason to select these relations? Can we predict other relations using the same framework?\n\n3. The motivation of the KGC task is not clearly explained. If the geospatial information is given, can we directly use the tool to get their relations? The paper should provide more insights into the motivation behind KGC task and the necessity of using LLMs for it.\n\n4. The ablation study is not clearly described. The paper should provide more details on the datasets, settings, and the performance of the final model used in Table 10. Otherwise, it is hard to evaluate the effectiveness of each module.\n\n5. The iterative trajectory self-refinement module is not well explained. Based on what criteria is the trajectory refined? How many iterations are performed? How does the performance improvement as the number of iterations increases? Moreover, if LLM lacks the knowledge about urban KGs as the motivation of this paper, how can it be a good judger? Is there any analysis about the correctness of the comments and refined trajectory?\n\n6. The cost for fine-tuning data construction is not discussed. How much data is required for fine-tuning the model? What is the cost of collecting and labeling the data for fine-tuning?\n\n7. In experiments, the paper should compare UrbanKGent with existing UrbanKG construction methods rather than purely focusing on the LLMs which are not tailored for this task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1&Q1\n\nDue to the tables related to weakness 1 and question 1 are included in the Supplementary PDF of the **Author Rebuttal**, we have moved the corresponding responses to the **Author Rebuttal** for better understanding. We apologize for any inconvenience this may cause.\n\n> W2,W3&Q2\n\nWe choose geospatial relations for KGC tasks for two reasons. First, spatial relations are the majority in UrbanKG (about 60% in both datasets, as reported in Table 15 of PDF). Second, as validated by recent works [1-2], spatial relation understanding is one of the most challenging tasks for LLMs. Therefore, we use geospatial relations to demonstrate the effectiveness of the UrbanKGent in KGC. Other relations, including temporal and functional, can be predicted using this framework by extending corresponding instructions. For example, by injecting the time information (e.g., the date of building was built) of urban entities into the instruction, we can complete their missing temporal relation (e.g., built earlier than). The capability of UrbanKGent to identify other relation types is also echoed by its success in RTE tasks.\n\nRegarding external tool use, it is possible to invoke tools to calculate certain geospatial relations, but not all, with sufficient geospatial information. For example, given the Region Connection Calculus (RCC) of two entities, we can derive up to 5 defined geospatial relations using the GIS system. However, some spatial relations may not be able to be extracted using tools, especially when entity information is incomplete, or the corresponding relation type and tool are unknown. Overall, we acknowledge the merit of using external tools to derive urban relations. In fact, we have devised tool use as an indispensable block in the tool-augmented trajectory refinement block, and are working to incorporate more spatiotemporal tools to improve the effectiveness of our framework.\n\nReference: \n\n[1] GeoLM. EMNLP. 2023. [2] Are Large Language Models Geospatially Knowledgeable? 2023.\n\n> W4&Q4\n\nWe use UrbanKGent-7B as the final model, with the same setting in the overall experiment (Section 5.2) on the NYC dataset. For the ablation study (Line 720 - 724), we remove the knowledgeable instruction template ($UrbanKGent-7B^{\\spadesuit }$), multi-view design ($UrbanKGent-7B^{\\star }$), external geospatial tool invocation ($UrbanKGent-7B^{\\ddagger}$), and iterative trajectory self-refinement ($UrbanKGent-7B^{\\dagger}$) respectively from UrbanKGent-7B to validate their effectiveness. The results show that removing any of these modules will lead to performance degeneration, demonstrating their effectiveness. We will add more details in E.2 and provide more descriptions in the caption of Table 10.\n\n> W5&Q3\n\nWe prompt the backbone LLM to automatically judge if a trajectory is faithful and provide refinement feedback for the unfaithful trajectory, more details are explained in **Lines 272-274**. Trajectory Updater will then follow the feedback to refine the current trajectory. For the maximum iterations, as explained in **Lines 280-282**, we set it to 3 to avoid excessive cost. To further understand the role of iteration numbers, we set maximum iterations from 0 to 10, and report the model's performance as well as the average stopping epochs in which the predefined stopping condition is satisfied (i.e., all trajectories are faithful).\n\n|The GPT-evaluation results on NYC dataset using UrbanKGent-7B||||\n-|-|-|-\nmaximum Iteration|stopping epochs|RTE (acc)|KGC (acc)\n0|-|0.44|0.47\n3|2.46|0.46|0.49\n5|2.84|0.47|0.48\n10|3.25|0.48|0.49\n\nAs can be seen, the model performance drops without iterations. However, larger than 3 iterations only lead to marginal improvements. This suggests that 3 iterations are cost-effective for the model to achieve the predefined stopping condition.\n\nMoreover, we agree with the reviewer that LLM cannot work as a good judger if it lacks urban knowledge. As shown in Table 10, the $UrbanKGent-7B^{\\spadesuit }$, whose knowledgeable instruction template (Line 720-721) is removed, performs poorly, although it incorporates the iterative trajectory self-refinement method. Regarding the correctness of the refined trajectory, the performance improvement reported in the above table indicates that the refined trajectory is more accurate. The reported case study in Figure 8 in our paper also echos its effectiveness, e.g., the model can identify some missing triplets after self-refinement.\n\n> W6&Q5\n\nThe constructed instruction dataset for fine-tuning can be found in **Supplementary Material**. As suggested, we provide statistics and the cost of prompting GPT-4 for instruction generation in the following Table:\n\nTask|Number of raw records|Number of instructions generated by UrbanKGent pipeline|Cost (dollar)\n-|-|-|-\nRTE|354|4,246|12.51\nKGC|354|2,011|18.07\nIn total|708|6,257|30.58\n\nAs can be seen, 6,257 instructions are used and the cost of calling GPT-4 is 30.58 dollars.\n\n> W7&Q6\n\nExisting UrbanKG construction methods rely heavily on manual extraction of urban entities and relations, as detailed in **Lines 394-400**. These manual methods, while effective, are not directly comparable to our automated approach without corresponding entities and relations annotated in the training data.\n\nHowever, we agree that a comparison can still provide valuable insights. To this end, we performed a quantitative comparison between previous manual methods and our proposed automatic paradigm using the latest UrbanKG benchmark [1]. As shown in **Table 4** in our paper, UrbanKGent can construct UrbanKGs with the same scale of triplets and entities in the benchmark by using only one-fifth of the data. Furthermore, our method expands the relationship types by a hundred times, demonstrating improvements in efficiency and comprehensiveness.\n\nThis comparison underscores UrbanKGent's potential for automatic UrbanKG construction, making it a competitive alternative against manual methods.\n\nReference: \n\n[1] UUKG. NeurIPS 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iXsnz3n1Ax", "reviewer": "Reviewer_rXu6", "summary": "The paper introduces UrbanKGent, a comprehensive framework utilizing a large language model (LLM) for constructing urban knowledge graphs (UrbanKGs). The key components of UrbanKGent involve creating an instruction set for tasks like relational triplet extraction and knowledge graph completion, which are tailored to urban data's heterogeneity and spatial characteristics. An iterative refinement module is employed to improve and adjust trajectories derived from the GPT-4 model, enhancing the quality of the knowledge graph. The framework is then fine-tuned using these enhanced trajectories on the Llama 2 and Llama 3 models, resulting in UrbanKGC agents, available in 7/8/13B versions.", "questions": "1. As mentioned in Appendix Section D.1, human annotators are employed to fill out an evaluation form. Is it possible to provide more detailed information on human annotators, such as an exact number of human annotators?", "limitations": "As acknowledged in appendix section F, the paper has limitations on the further application demonstration of construction UrbanKGs and the evaluation method in this work is cost-intensive.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Intensive efforts have been dedicated including crawling and preprocessing raw data, empirically validating the insufficiency of GPT-4, and instruction fine-tuning the UrbanKGC agent.\n2. The paper focuses on real-world scenarios with domain knowledge infused in UrbanKGC agent construction and carries out comprehensive evaluations on real-world datasets using human and GPT-4 self-assessment.", "weaknesses": "1. The computational efficiency of the proposed algorithms and their scalability with the size of the dataset is not thoroughly discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> [W1] The computational efficiency of the proposed algorithms and their scalability with the size of the dataset is not thoroughly discussed.\n\nWe sincerely appreciate the reviewer's valuable comment, which helps us improve the quality of our paper. As suggested, we provide detailed inference latency for the UrbanKGent family across different dataset scales in the following table. As shown in Table 1, by applying VLLM [1] techniques to accelerate our current framework, UrbanKGent-13B can automatically construct an UrbanKG with approximately one million triples (905,442 extracted from the NYC-Large dataset) in about 62 minutes. We also report the average inference time of the UrbanKGent family. For every 1,000 data records processed, the UrbanKGent takes 0.57 minutes on the 7B version, 0.76 minutes on the 8B version, and 1.53 minutes on the 13B version, respectively.\n\nTable 1: The inference latency comparison of UrbanKGC using the UrbanKGent family before and after using VLLM acceleration. We use two middle-size datasets (i.e., NYC and CHI) and two large-scale datasets (i.e., NYC-Large and CHI-Large) for UrbanKG construction. The accelerated inference latency is **bolded.**\n\nDataset|Latency (minutes)|||Data Volume\n-|-|-|-|-\n|| UrbanKGent-7B     | UrbanKGent-8B    |UrbanKGent-13B|\nNYC|6.83/**1.19**|7.62/**1.58**|16.48/**3.19**|2,089\nCHI|3.60/**0.62**|4.02/**1.13**|8.69/**1.68**|1,102\nNYC-Large|132.33/**23.07**|147.75/**30.76**|319.38/**61.93**|40,480\nCHI-Large|94.39/**16.55**|105.36/**21.93**|227.76/**44.47**|28,868\n-|3.27/**0.57**|3.65/**0.76**|7.89/**1.53**|Per 1,000 records\n\nCurrently, the maximum dataset scale we used is about forty thousand. We are working on processing larger-scale data and analyzing potential scalability issues.\n\n**Reference**:\n\n[1] Kwon, Woosuk, et al. \"Efficient memory management for large language model serving with pagedattention.\" *Proceedings of the 29th Symposium on Operating Systems Principles*. 2023.\n\n> [Q2] As mentioned in Appendix Section D.1, human annotators are employed to fill out an evaluation form. Is it possible to provide more detailed information on human annotators, such as an exact number of human annotators?\n\nThanks to the reviewer's valuable suggestion. We invite 3 AI experts to fill out the evaluation form of the prediction results from various LLM variants. All of them possess work experience as algorithm engineers at Internet or AI companies. To avoid potential performance bias (as a priori, it is believed that larger-size LLMs are better), we do not reveal which results come from which LLMs. This rigorous process resulted in the annotation of 200 random samples. We will provide a more detailed description in Appendix Section D.1 in our final paper version. We sincerely appreciate the valuable suggestions provided by the reviewer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IknHSSRhrj", "reviewer": "Reviewer_e3YZ", "summary": "This paper presents a framework denoted UrbanKGent, for finetuning an LLM to assist the construction of knowledge graph construction (specifically triplet extraction relation prediction). The gist of the study is to construct an ad-hoc corpus for finetuning. In this process, a method of trajectory refinement is proposed to enhance both the effectiveness and explainability of the framework.", "questions": "See weaknesses.", "limitations": "See weaknesses.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1) An interesting and important application domain of LLM\n2) A generally well-developed workflow for tackling the target problem\n3) The evaluation is more or less adequate for showing the superiority", "weaknesses": "1) It seems that the generation of the corpus, which is a key ingredient of the framework, lacks a systematic and scientific methodology for its generation.\n2) The transferability is questionable – is this method useful in cities that are not finetuned?\n3) The description of the roles of GPT4 and the smaller LLM that is finetuned is vague.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> [W1] It seems that the generation of the corpus, which is a key ingredient of the framework, lacks a systematic and scientific methodology for its generation.\n\nSorry for the confusion. As mentioned by the reviewer, the corpus is very important for the UrbanKGent framework. Therefore, we construct the corpus by uniform sampling (Line 308-310) from multi-source urban data (i.e., AOI, road network, POI, review, and web page), which are widely used for urban knowledge distillation. By finetuning with the generated corpus, we demonstrate the performance of LLMs on UrbanKGC tasks could be largely improved. We will provide a more clear description of the corpus generation in our final paper version to avoid potential confusion.\n\n> [W2] The transferability is questionable – is this method useful in cities that are not finetuned?\n\nThanks to the reviewer's question to help us clarify potential confusion. The proposed UrbanKGent framework could enhance UrbanKGC performance in both fine-tuning and non-fine-tuning scenarios. Specifically, as demonstrated in Table 3 of our paper, the performance of various LLM backbones consistently improves only using the UrbanKGent Inference pipeline without fine-tuning, compared with zero-shot reasoning and In-context-learning paradigms. Therefore, in cities where the LLM has not been fine-tuned, our method retains practical usage value.\n\n> [W3] The description of the roles of GPT4 and the smaller LLM that is finetuned is vague.\n\nSorry for the confusion. As mentioned in Section 4.3 (Line 286-288) in our paper, we derive GPT-4 for trajectory generation consisting of two steps (i.e., the instruction generation and iterative trajectory refinement shown in Figure 4). The obtained trajectory will be further used to fine-tune the small LLMs. The fine-tuned smaller LLMs then will be used for UrbanKGC tasks with faster inference speed and lower cost.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**Dear Reviewer GYCX**:

We thank you for the precious review time and valuable comments. To ease understanding, we have moved weakness 1 and question 1 here, which are related to UrbanKG entity and relation ontology statistics. Please refer to Table 13, Table 14, and Table 15 in the **PDF** for detailed information on UrbanKG entity and relation ontology statistics.

> [W1] This paper focuses on urban knowledge graph construction tasks. However, the details of the urban knowledge graph are not clearly defined. For example, the ontology of urban KGs, entity types, and relation types are not discussed. This limits the understanding of the proposed framework for readers who are not familiar with urban knowledge graphs.

> [ Q1] What is the ontology of urban KGs?

We apologize for the confusion. The constructed UrbanKGs can be found in **Supplementary Material**. As the reviewer suggested, we provide entity and relation ontology statistics in the PDF. As shown in Table 13, both urban entity and relation can be pre-categorized into 4 coarse-grained ontologies: spatial, temporal, functional, and others. After multi-view entity recognition and relation extraction (shown in Figure-4(a) in our paper), the fine-grained entity ontologies (1,028 and 755 entity types of NYC-Large and CHI-Large UrbanKGs, respectively) and fine-grained relation ontologies (2,138 and 1,366 relation types of NYC-Large and CHI-Large UrbanKGs, respectively) are obtained. To ease understanding, we also provide illustrative entity ontology (e.g., University) and relation ontology (e.g., Locate-in) examples in Table 14 and Table 15, respectively. In addition, we also display the entity and relation ontology distribution of constructed UrbanKGs on the NYC-Large and CHI-Large datasets. More detailed ontology information can be found in the Supplementary Material. 

We will include detailed statistics about the UrbanKG ontology in the final version of our paper. We sincerely appreciate the reviewer's valuable suggestions.

Best,

NeurIPS 2024 Conference Submission 16941 Authors

Meta-review: Existing Urban Knowledge Graph Construction (UrbanKGC, which encompass relation extraction and KG completion tasks) approaches either rely on manually designed patterns/rules or annotated corpus to train LLMs. This work proposes  an agent framework UrbanKGent to solve UrbanKGC with minimal human labor and has the following components:
1) prescripted LLM process prompts:
a) triple extraction prompts based on a two-turn dialog with LLM
b) KG completion prompts infused with geospatial information
c) toolkit description/invocation prompts
d) reasoning refinement prompts
2) iterative trajectory refinement based on a verifier/critique prompt and an updater prompt
3) to achieve efficient inference smaller LLMs are fine-tuning on the larger LLMs trajectories. 

Evaluation on two real-world datasets using both human and GPT-4 self-evaluation show that UrbanKGent outperforms various LLM baselines and end-to-end models.

Reviewers recognize the importance of the application and the motivation behind the system design. This is solid work applying LLM to information extraction in the UrbanKG domain, and may set the stage for more advanced learning approaches in the future.
Decision: Accept (poster)