SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yansong Ning $^{1}$ , Hao Liu $^{1,2*}$ , Hao Wang $^{3}$ , Zhenyu Zeng $^{3}$ , Hui Xiong $^{1,2}$
<sup>1</sup> AI Thrust, The Hong Kong University of Science and Technology (Guangzhou)
$^{2}$  CSE, The Hong Kong University of Science and Technology
3 Alibaba Cloud Intelligence Group
yning092@connect.hkust-gz.edu.cn liuh@ust.hk
cashenry@126.com zhenyu.zzy@alibaba-inc.com xionghui@ust.hk
Accurate Urban SpatioTemporal Prediction (USTP) is of great importance to the development and operation of the smart city. As an emerging building block, multi-sourced urban data are usually integrated as urban knowledge graphs (UrbanKGs) to provide critical knowledge for urban spatiotemporal prediction models. However, existing UrbanKGs are often tailored for specific downstream prediction tasks and are not publicly available, which limits the potential advancement. This paper presents UUKG, the unified urban knowledge graph dataset for knowledge-enhanced urban spatiotemporal predictions. Specifically, we first construct UrbanKGs consisting of millions of triplets for two metropolises by connecting heterogeneous urban entities such as administrative boroughs, POIs, and road segments. Moreover, we conduct qualitative and quantitative analysis on constructed UrbanKGs and uncover diverse high-order structural patterns, such as hierarchies and cycles, that can be leveraged to benefit downstream USTP tasks. To validate and facilitate the use of UrbanKGs, we implement and evaluate 15 KG embedding methods on the KG completion task and integrate the learned KG embeddings into 9 spatiotemporal models for five different USTP tasks. The extensive experimental results not only provide benchmarks of knowledge-enhanced USTP models under different task settings but also highlight the potential of state-of-the-art high-order structure-aware UrbanKG embedding methods. We hope the proposed UUKG fosters research on urban knowledge graphs and broad smart city applications. The dataset and source code are available at https://github.com/usail-hkust/UUKG/.
Urban SpatioTemporal Prediction (USTP) aims to forecast future urban dynamics by simultaneously capturing the spatial and temporal autocorrelations from past observations. Recently, with the development of machine learning technologies and the collection of large-scale urban data, USTP has achieved remarkable progress in various urban predictive tasks, such as traffic management, pollution monitoring, and emergency response [1, 2, 3]. USTP has become an essential service of the modern smart city.
In prior literature, many efforts have been devoted to improving the USTP performance by exploiting latent knowledge from diverse urban data sources [4]. In particular, one commonly used approach is manually extracting features from different datasets to integrate additional information.
For example, Tompson et al. [5] and Chen et al. [6] customize urban features from external dataset for crime prediction and bike trip prediction. However, these approaches heavily rely on a deep understanding of the application domain and are labor-intensive. Recently, inspired by the success of the Knowledge Graph (KG) in natural language processing tasks, the urban knowledge graph has been adopted to improve USTP. For instance, Wang et al. [7]
Figure 1: The generation process of UUKG.
construct a dedicated spatiotemporal knowledge graph by regarding trajectory and timestamp as entities to improve trajectory prediction. Liu et al. [8] construct user check-in relations to help mobility prediction. However, existing UrbanKGs are task-specific and none of them is publicly available, which discourages researchers from adopting it for their own work and thus limits the flourishing of knowledge-enhanced urban spatiotemporal prediction.
Therefore, an open-sourced and multifaceted urban knowledge graph dataset compatible with various USTP tasks is necessary to be proposed. It is a non-trivial problem due to the following two challenges. (1) How to construct unified urban knowledge graphs? The urban data collected from different devices and providers describe the urban system from different aspects and granularities. They are usually disjoint datasets with different spatiotemporal ranges, which cannot be directly joint utilized for USTP. It is appealing to extract and align heterogeneous urban knowledge in a unified graph organization to satisfy diverse downstream USTP requirements. (2) How to preserve complicated structural urban knowledge? Existing UrbanKG-driven USTP approaches simply adopt general KG embedding methods or design complex task-specific module to project symbolic entities and relations into low-dimensional embeddings, which fails to preserve unique structural patterns in the urban knowledge graph. As illustrated in Figure 2, the urban knowledge graph includes diverse structures such as hierarchy and cycle. It is crucial to capture such high-order semantic knowledge to empower downstream spatiotemporal prediction tasks.
To address the aforementioned challenges, in this study, we present an Unified Urban Knowledge Graph (UUKG) dataset for knowledge-enhanced urban spatiotemporal predictions. Figure 1 illustrates the workflow of UUKG construction. For a given city, we first construct an Urban Knowledge Graph (UrbanKG) from multi-sourced urban data. By extracting and organizing entities (e.g., POIs, road segments, etc.) into a multi-relational heterogeneous graph, UrbanKG encodes various high-order structural patterns in a unified configuration (i.e., a multi-scale spatial hierarchy), which facilitates joint processing for various downstream USTP tasks. Moreover, we qualitatively and quantitatively analyze these diverse high-order structures (i.e., hierarchies and cycles in Figure 2) to guide us in using tailored KG embedding methods to derive effective and generalizable knowledge representations. By learning structure-aware embeddings of entities and relations using state-of-art non-Euclidean space embedding models (e.g., modeling hierarchies in hyperbolic space and capturing cycles in spherical space), the high-order structure information could be preserved in a proper way.
Finally, we conduct comprehensive experiments to benchmark diverse urban tasks, including the UrbanKG completion task, three urban flow prediction tasks, and two urban event prediction tasks. The empirical studies not only validate the effectiveness of the unified urban knowledge graph for improving various USTP tasks, but also uncover the high-order structures within UrbanKG, with proper modeling, can further strengthen the urban spatiotemporal prediction performance.
- We propose UUKG, the first open-source UrbanKG dataset for knowledge-enhanced urban spatiotemporal predictions. As a pilot study, we envision our experiences and results offering exciting opportunities to advance previous USTP methods.
- We demonstrate the importance of urban high-order structure modeling for urban knowledge representation and investigate to use structure-aware KG embedding methods to capture them effectively.
- Extensive experiments on KG completion and five USTP tasks demonstrate the effectiveness of the constructed UrbanKG and verify modeling high-order structures is beneficial to urban spatiotemporal prediction.
Table 1: The statistics of raw datasets.
Dataset	Description	Sample format	Records
New York	Chicago
Administrative division	Boundary	[“New Yorck”, range(40.50, 40.91, -74.25, -73.70)]	1	1
# of Borough	[“Queens”, polygon(40.54 -73.96, ...)]	5	6
# of Area	[“Jamaica”, “Queens”, polygon(40.69 -73.82, ...)]	260	77
Road network	# of Segment	[road id, road name, start junction, end junction, type, line range]	110,919	71,578
# of Category	[191751, “Queens Boulevard”, 59378, 4798, tertiary, line(40.73 -73.82, ...)]	5	5
# of Junction	[junction id, junction type, coordinate]	62,627	37,342
# of Category	[59378, crossing, coordinate(40.78 -73.98)]	5	6
POI	# of POI	[poi id, poi name, poi type, coordinate]	62,450	31,573
# of Category	[34633854, “Empire State Building”, corporation, coordinate(40.75, -73.99)]	15	15
2 UrbanKG Construction
We first introduce the datasets that will be used for urban knowledge graph construction and data preprocessing details, then we present the construction of urban knowledge graphs.
2.1 Data Collection and Preprocessing
We acquire urban knowledge for two large cities, New York and Chicago, from three data sources. Table 1 summarizes the statistics of the datasets.
Administrative Division data. The administrative division describes multi-level spatial entities that the government uses to make administrative decisions, which contains rich geographical information for USTP task [9, 10]. In this work, we collect three administrative divisions, i.e., city, borough, and area. We first manually define the rectangular latitude and longitude boundaries for each city (e.g., New York and Chicago). Then we collect borough and area data from NYC Gov  ${}^{2}$  and CHI Gov  ${}^{3}$  , the official websites of New York and Chicago. Each borough record contains a borough name and polygon boundaries. Each area record contains an area name, corresponding borough name, and the polygon boundaries. Example in Table 1 is a record of "Jamaica" area in the "Queens" borough.)
Road Network Data. The road network provides rich topology knowledge of the transportation system, which plays a critical role in various USTP tasks (e.g., traffic congestion prediction [11] and traffic incident detection [12]). We utilize the rectangular of each city to query the corresponding road network data (road segments and road junctions) from Open Street Map (OSM<sup>4</sup>). Each road segment record contains a unique id, a road name, a start road junction id, a end road junction id, and line geographical range. Each road junction record contains a unique id, junction type and coordinate (latitude and longitude). Example in Table 1 is a road record of "Queens Boulevard", and it starts from junction 594778 and ends at junction 4798.
POI Data. POI data contains urban contextual semantics, which have been widely adopted as auxiliary features to enhance USTP tasks (e.g., traffic prediction[13] and crime prediction [14]). We collect POI data from OSM. We utilize the boundary of each city from the administrative division data to query the corresponding POI records through the open API provided by OSM. Each POI record consists of a unique id, a POI name, POI category and the coordinate. Example in Table 1 is a POI record where (40.75, -73.99) is the location and the POI is a corporation.
Before constructing the UrbanKG, we first preprocess the raw datasets. We filter out POIs and road networks that don't belong to any administrative borough or area in each city. Besides, we merge minority POI categories (e.g., grandstand and canopy) to avoid potential long-tail issues. As records from different data sources are disjoint, we align each record according to the administrative areas.
2.2 Knowledge Graph Construction
Then we introduce the process of constructing the urban knowledge graph (UrbanKG).
Definition 1 UrbanKG. The UrbanKG is defined as a multi-relational graph  $\mathcal{G} = (\mathcal{E},\mathcal{R},\mathcal{F})$ , where  $\mathcal{E}$ ,  $\mathcal{R}$  and  $\mathcal{F}$  is the set of urban entities, relations and facts, respectively. In particular, facts are
$^{2}$ https://www.nyc.gov/
<sup>3</sup>https://www.chicago.gov/
Table 2: Major relations in UrbanKG.
Relation	Symmetric	Head & Tail Entity	Abbrev.	Relation	Symmetric	Head & Tail Entity	Abbrev.
POI Locates at Area	×	(POI, Area)	PLA	Junction Belongs to Road	×	(Junction, Borough)	JBR
Road Locates at Area	×	(Road, Area)	RLA	Borough Nearby Borough	✓	(Borough, Borough)	BNB
Junction Locates at Area	×	(Junction, Area)	JLA	Area Nearby Area	✓	(Area, Area)	ANA
POI Belongs to Borough	×	(POI, Borough)	PBB	POI Has POI Category	×	(POI, PC)	PHPC
Road Belongs to Borough	×	(Road, Borough)	RBB	Road Has Road Category	×	(Road, RC)	RHRC
Junction Belongs to Borough	×	(Junction, Borough)	JBB	Junction Has Junction Category	×	(Junction, JC)	JHJC
Area Locates at Borough	×	(Area, Borough)	ALB	-	-	-	-
Figure 2: An illustrative example of the urban knowledge graph with diverse hierarchical and cyclic structure. There are four types of entities (Borough, Area, POI and POI Category (PC)) and five types of relations (BNB, ALB, ANA, PLA and PHPC).
defined as  $\mathcal{F} = \{\langle h,r,t\rangle \mid h,t\in \mathcal{E},r\in \mathcal{R}\}$ , where each triplet  $\langle h,r,t\rangle$  describes that head entity  $h$  is connected with tail entity  $t$  via relation  $r$ . For example,  $\langle \text{Queen}, \text{Nearby}, \text{Brooklyn} \rangle$  represents the fact that Queen and Brooklyn are geographically adjacent.
The UrbanKG encodes diverse urban semantic knowledge such as the multi-level spatial adjacency (e.g., the Hammels area in Queens borough) and ontology (e.g., category of POIs and roads). We detail the entity and relation extraction below.
2.2.1 Entity Extraction
We extract 8 types of entities for UrbanKG: (1) Administrative Borough (Borough). Borough describes high-level administrative boundaries of a city. For example, New York has five boroughs: Queen, Bronx, Brooklyn, Manhattan, and Staten Island. (2) Functional Area (Area). Area refers to the subdivision of a city according to its function, such as residential, industrial, and commercial areas. (3) Point-Of-Interest (POI). A POI is a basic functional unit and venue. For example, cinemas and hospitals are two common types of POIs and they are the places people often check in. (4) Road Segment (Road). Road is the key element in the city and it forms the traffic network, which provides a reference and basis for human mobility. (5) Road Junction (Junction). Junction is where two or more road segments intersect. It is important for the urban road network as they have a great impact on traffic capacity and safety. (6) POI Category (PC). POI category describes the property and function of POI, such as finance, catering and so on. We define 15 categories of POI including finance, parking area, shopping, catering, and so on. (7) Road Category (RC). Road category describes the features of the road segment, such as motorway, trunk and so on. We preserve the six-types of most frequent road segments including motorway (expressway or river-crossing tunnels), primary traffic road, secondary traffic road, tertiary traffic road, residential traffic road, and trunk (branch roads such as expressway outbound bypass roads). (8) Junction Category (JC). Junction category helps distinguish several special types of road junctions, such as roundabout. We keep five types of road junction including motorway junction (road junction in expressway or river-crossing tunnel), traffic signal (road junction having traffic light), turning circle (road junction which is a roundabout), stop (road junction having stop signal) and crossing (road junction with no special type).
We report the detailed statistics of entities in Appendix A.1.
2.2.2 Relation Construction
As shown in Table 2, we define 13 relations for UrbanKG:
Table 3: Statistics of two UrbanKGs. We report graph hyperbolicity values  $\delta$  (always greater than or equal to zero while lower means more hierarchical) and the number of cycles.
Dataset	# Entity	# Relation	# Triplet	# Train	# Valid	# Test	# Cycle	# Hyperbolicty
NYC	236,287	13	930,240	837,216	46,512	46,512	1,090,884	δ = 0
CHI	140,602	13	564,400	507,960	28,220	28,220	532,108	δ = 0
Geographic Inclusion. This type of relation describes the geographical inclusion relations between entities. We extract 5 geographical inclusion relations for UrbanKG: (1) POI/Road/Junction Locates at Area (PLA/RLA/JLA). Through these relations, we can obtain the fact between Area and POI/Road/Junction. For example, triplet  $\langle$  POI 32, PLA, Area 75  $\rangle$  indicates that POI 32 is located in Area 75. (2) POI/Road/Junction Belongs to Borough (PBB/RBB/JBB). Through these relations, we can obtain the fact between Borough and POI/Road/Junction. (3) Area Locates at Borough (ALB). Through this relation, we can obtain the geographic inclusion relations between Area and Borough. For example, triplet
<Area 75, ALB, Borough I> explains that Area 75 belongs to Borough 1. (4) Junction Belongs to Road (JBR). Through this relation, we can describe the Junction on which Road, thus could preserve the topology of the road network.
Figure 3: An illustrative visualization of the four UrbanKG entities in NYC and CHI.
Geographic Aadjacency. This type of relation describes the geographical adjacency relations between entities. We extract 2 relations of this type for UrbanKG: (1) Borough Nearby Borough (BNB). Through this relation, we can obtain the adjacency fact between Boroughs. (2) Area Nearby Area (ANA). Through this relation, we describe the adjacency between Areas.
Category. This type of relation connects entities to the category they belong to. We extract three category relations for UrbanKG: (1) POI Has POI Category (PHPC). (2) Road Has Road Category (RHRC). (3) Junction Has Junction Category (JHJC). For example, triplet  $\langle POI 32, PHPC, PC 4 \rangle$  indicates POI 32 belongs to the category PC 4.
2.2.3 UrbanKG Management
Following the above process, we obtain UrbanKGs for two large cities as shown in Table 3, i.e., New York (NYC) and Chicago (CHI). To serve the large-scale knowledge graph with millions of triplets, we adopt Neo4j, a graph database system, to store, query, and update the UrbanKG. The data privacy issue is critical in smart city applications. In this study, the dataset we use doesn't include any user-level information. However, recent studies prove that sensitive individual information can be inferred from a few location check-ins [15]. Thus, we round the coordinate of each record to 100 meters to avoid potential information leakage when incorporating UrbanKG with other urban spatio-temporal data.
2.3 Visualization and Structural Analysis
As shown in Figure 3, the multi-level UrbanKG contains entities providing administrative division information (Borough & Area), traffic topology (Road network), and urban function distribution (POI), and they together form rich semantic information (i.e., High-order hierarchy and high-order cycle) about the city.
High-order Hierarchy. Hierarchical information is ubiquitous in capturing knowledge from KG [16, 17, 18] since much knowledge is commonly organized hierarchically. For example, the hierarchy (Borough->Area->POI->PC) in Figure 2 organizes urban knowledge in a hierarchical way. Embracing and modeling such hierarchy empowers to uncover deeper semantics, e.g., we can infer the main function of an area from the POI quantities and categories in this area. As shown in Table 3, we also quantitatively find the hyperbolicities [19] of two UrbanKGs are zero, which means that UrbanKGs
are not significantly different from a tree in terms of their geometric properties [20, 21]. Thus they are suitable to be represented in hyperbolic space [22, 19], where hierarchies can be losslessly embedded.
High-order Cycle. Cycle modeling is beneficial to improve knowledge representation [23, 24, 25] as it implies rich semantics of graphs. We find their existence in constructed UrbanKG by analyzing the topological structure. For example, the homogeneous cycle and heterogeneous cycle in Figure 2 uncover Borough geographic semantics and the geographic semantics of Area and Borough, respectively. Moreover, Table 3 counts the number of cycles in the two UrbanKG to further verify our analysis. Unlike hierarchy, these cycles are suitable to be modeled in spherical space [22, 26].
The qualitative and quantitative analysis reveal diverse high-order structural patterns in UrbanKG, and thus inspires us to leverage state-of-the-art high-order structure-aware knowledge graph embedding methods to extract tailored and comprehensive urban knowledge. For instance, UrbanKG can be embedded into hyperbolic space to capture hierarchies or into spherical space to model cycles.
3 Structure-aware UrbanKG Embedding
This section investigates the impact of different structure-aware embedding methods (e.g., the hyperbolic space embedding module to capture hierarchy) on UrbanKG representation learning. We first give problem formulation of UrbanKG embedding and introduce how to utilize structure-aware embedding methods to represent those high-order structures within UrbanKG.
Given the constructed UrbanKG  $\mathcal{G} = (\mathcal{E},\mathcal{R},\mathcal{F})$ , we aim to learn low-dimensional entity embeddings  $e_{\mathcal{E}}$  and relation embeddings  $e_{\mathcal{R}}$ , so that diverse high-order structure knowledge can be preserved to benefit various downstream USTP tasks.
We compare the performance gap of different structure-aware knowledge graph embedding methods to investigate which approaches are suitable for modeling the unique structures in UrbanKG. The quality of UrbanKG embedding is evaluated using the link prediction task [27], which aims to predict the missing head or tail entity for a triplet  $\langle h, r, t \rangle$ . For each UrbanKG in Table 3, we follow standard data augmentation protocol [28] by adding inverse relations.
Evaluation Metrics. We report two popular metrics: (1) Mean reciprocal rank (MRR), the mean of inverse of correct entity ranking [29]. (2) Hits@K  $(\mathrm{K} = 1,3,10)$ , the percentage of correct entities in top-K ranked entities [30, 31]. Note we filter out true triplets appearing in the training set during evaluation [30], because predicting a low rank for those triplets should not be penalized.
Methods. We utilize classical Euclidean methods as baselines and choose state-of-art structure-aware non-Euclidean approaches as backbones.
- Euclidean Models: (1) TransE [30] is a classical translational model; (2) DisMult [32] is a matrix factorization model; (3) ComplEx [33] is an extension of DisMult in complex space; (4) RotatE [34] is a rotation-based method in complex space; (5) MuRE [16] is a translational distance method with a diagonal relational matrix. (6) TuckER [35] is a tensor decomposition method. (7) QuatE [36] is a hypercomplex space embedding method.
- Non-Euclidean Models: (1) MuRP/MuRS [16] are hyperbolic or spherical model with diagonal relational matrix; (2) RotH/RefH [17] are hyperbolic embedding method with rotation or refelection. (3) AttH [17] is a hyperbolic embedding method combining rotation and reflection; (4) ConE [18] is a hyperbolic embedding method with transformation between hyperbolic cone; (5) M2GNN[26] is a GNN-based product space embedding model which can model cycles and hierarchies at the same time. (6) GIE [37] is a product space model considering geometry interaction.
Implement Details. We implement all models using PyTorch. All experiments are conducted on eight NVIDIA RTX 3090 GPUs. The negative sampling size is fixed to 50. We conduct hyperparameters grid search by early stop on the validation sets and we report details in Appendix A.2.
Due to the large scale of our UrbanKG, we didn't perform Graph Neural Updater when reproducing.
Table 4: Overall link prediction results on NYC and CHI dataset. We utilize E, C, S, H, P to denote the Euclidean, Complex, Spherical, Hyperbolic and Product space, respectively. Best results in each space are underlined.
Type	Space	Model	NYC	CHI
MRR	Hits@10	Hits@3	Hits@1	MRR	Hits@10	Hits@3	Hits@1
Euclidean models	E	TransE	.507	.563	.528	.470	.485	.556	.501	.436
E	DisMult	.401	.478	.433	.355	.395	.479	.469	.394
E	MuRE	.516	.613	.545	.468	.493	.601	.536	.437
E	TuckER	.513	.609	.541	.466	.488	.584	.525	.423
C	RotatE	.274	.363	.309	.220	.306	.385	.336	.258
C	ComplEx	.259	.357	.305	.195	.304	.385	.337	.253
C	QuatE	.321	.388	.347	.282	.396	.490	.427	.345
Non-Euclidean models	S	MuRS	.528	.622	.552	.478	.501	.619	.536	.437
H	MuRP	.545	.635	.570	.497	.519	.634	.555	.456
H	RotH	.526	.601	.550	.472	.511	.626	.548	.447
H	ReFH	.524	.610	.549	.473	.509	.629	.542	.451
H	ATTH	.539	.603	.556	.503	.514	.610	.542	.463
H	ConE	.542	.629	.563	.485	.513	.617	.535	.465
P	M2GNN	.561	.638	.578	.521	.540	.651	.571	.481
P	GIE	.573	.665	.600	.523	.552	.660	.580	.498