SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xu Liu $^{1}$ , Yutong Xia $^{1}$ , Yuxuan Liang $^{2*}$ , Junfeng Hu $^{1}$ , Yiwei Wang $^{1}$ , Lei Bai $^{3}$ , Chao Huang $^{4}$ , Zhenguang Liu $^{5*}$ , Bryan Hooi $^{1}$ , Roger Zimmermann $^{1}$
$^{1}$ National University of Singapore,  $^{2}$ Hong Kong University of Science and Technology (Guangzhou),
$^{3}$ Shanghai AI Laboratory,  $^{4}$ University of Hong Kong,  $^{5}$ Zhejiang University
{liuxu, junfengh, y-wang, bhooi, rogerz} @comp.nus.edu.sg, {yutong.x, yuxiang} @outlook.com,
{baisanshi, chaohuang75, liuzhenguang2008}  $@$  gmail.com
Road traffic forecasting plays a critical role in smart city initiatives and has experienced significant advancements thanks to the power of deep learning in capturing non-linear patterns of traffic data. However, the promising results achieved on current public datasets may not be applicable to practical scenarios due to limitations within these datasets. First, the limited sizes of them may not reflect the real-world scale of traffic networks. Second, the temporal coverage of these datasets is typically short, posing hurdles in studying long-term patterns and acquiring sufficient samples for training deep models. Third, these datasets often lack adequate metadata for sensors, which compromises the reliability and interpretability of the data. To mitigate these limitations, we introduce the LargeST benchmark dataset. It encompasses a total number of 8,600 sensors in California with a 5-year time coverage and includes comprehensive metadata. Using LargeST, we perform in-depth data analysis to extract data insights, benchmark well-known baselines in terms of their performance and efficiency, and identify challenges as well as opportunities for future research. We release the datasets and baseline implementations at: https://github.com/liuxu77/LargeST.
Road traffic forecasting plays a pivotal role in enhancing urban planning, traffic management, and public safety, making it one of the most critical components of Intelligent Transportation Systems [35]. In recent years, the burgeoning technique of machine learning, particularly deep learning, has exhibited remarkable advancements in this task [26]. Among the advances, spatio-temporal graph neural networks have demonstrated great promise and become the widely embraced tool for accurate traffic predictions [34, 19, 32, 1, 4]. Specifically, they utilize graph neural networks to capture spatial correlations among sensors and employ sequential models to model temporal dependencies.
What steps can we take to further advance research in traffic forecasting? Historically, high-quality and large-scale benchmark datasets have proven their value in driving research frontiers, as exemplified by the transformative impact of ImageNet [6] in computer vision, GLUE [31] in natural language processing, and OGB [13] in the general graph. In traffic forecasting research, however, we argue that commonly-used datasets present critical issues that may pose obstacles to future progress.
*Y. Liang and Z. Liu are the corresponding authors of this paper.
(a) Overview of the LargeST dataset
Figure 1: An illustration of the LargeST benchmark dataset.
(b) Fine-grained distribution of sensors
Issues of Existing Datasets. Most of the widely-used traffic datasets are limited in scale compared to real-world traffic networks. For example, widely-utilized benchmarks such as PeMS03, 04, 07, and 08 [30], comprise merely hundreds of nodes and edges (see Table 1 for details). However, the real-world traffic networks usually have much larger scales to be analyzed. For instance, California, US, alone possesses nearly 20,000 operational sensors. As traffic forecasting models are extensively developed on these small datasets, the majority of them turn out to be not scalable to larger sensor networks (see Section 5). Moreover, existing traffic datasets suffer from a dearth of temporal coverage, often spanning less than 6 months of data. This restricted duration hinders the study of long-term seasonal patterns and limits the number of training instances that can be used for deep model training. Another critical limitation of commonly-used datasets lies in the insufficient metadata available for individual nodes. Considering that traffic sensors represent specific points with tangible significance, the incorporation of node metadata in datasets is crucial. This inclusion serves to enhance the overall reliability of datasets and enables better interpretability of model predictions.
Contributions. In this work, we propose LargeST as a new benchmark dataset (see Figure 1), with the goal of facilitating the development of accurate and efficient methods in the context of large-scale traffic forecasting. The distinguishing characteristic of LargeST lies not only in its extensive graph size, encompassing a total of 8,600 sensors in California, but also in its substantial temporal coverage and rich node information - each sensor contains 5 years of data and comprehensive metadata. In addition to dataset construction, we conduct comprehensive data analysis, implement a suite of well-known traffic forecasting baselines, and perform extensive benchmark experiments. According to the empirical results, we summarize and highlight research challenges and future opportunities. LargeST is an open-source project hosted on GitHub. We will keep monitoring new research advancements in the field and summarize them in the repository.
This section commences by introducing the traffic forecasting task, followed by a brief review of existing efforts in this area.
The objective of traffic forecasting is to predict target attributes (e.g. traffic flow) in future steps based on historical observations over a directed sensor graph. To define the graph topology, the common practice [34, 19, 32] build the adjacency matrix  $\mathbf{A}$  using a thresholded Gaussian kernel [29], where  $A_{ij} = \exp \left(-\frac{d_{ij}^2}{\sigma^2}\right)$  if  $\exp \left(-\frac{d_{ij}^2}{\sigma^2}\right) \geqslant r$  else  $A_{ij} = 0$ . Here,  $d_{ij}$  denotes the road network distance between sensors  $i$  and  $j$ ,  $\sigma$  is the standard deviation of all distances, and  $r$  is the threshold.
2.2 Deep Learning-based Traffic Forecasting
In recent years, deep neural networks have emerged as the preferred method for traffic forecasting [19, 2, 22, 23, 7], thanks to their advanced learning capacity. They generally combine graph neural networks (GNNs) with either Recurrent Neural Networks (RNNs) or Temporal Convolutional Networks (TCNs) to capture intricate spatial and temporal dependencies in traffic data. For example, a pioneering work DCRNN [19] introduced a novel diffusion convolution that works alongside GRU. Other notable works such as ST-MetaNet [27], AGCRN [1], and DGCRN [18] have also employed RNNs and their variants. To improve training speed and leverage parallel computation, a plethora of approaches such as STGCN [34], GWN [32], and DMSTGCN [11] have replaced RNNs with dilated causal convolution for temporal patterns modeling. Moreover, attention mechanisms have been utilized in works like GeoMAN [20] and ASTGCN [10] to model spatial and temporal correlations. Recent developments in the field highlight two noteworthy trends. The first involves coupling GNNs with neural ordinary differential equations to produce continuous layers for modeling long-range spatio-temporal dependencies, e.g., STGODE [8] and STGNCDE [4]. The second trend involves building adjacency matrices at different time steps to capture the dynamic correlations among nodes, e.g., DGCRN [18], DSTAGNN [17], and  $\mathrm{D}^2\mathrm{STGNN}$  [28]. These approaches have exhibited promising performance on existing datasets, which, however, have limited size in terms of the number of nodes, edges, and time frames (see Table 1). This raises valid concerns about their scalability when applied to large-scale traffic forecasting datasets, which strive to encompass more realistic real-world scenarios.
3 Limitations of Existing Traffic Datasets
Table 1 presents a comparison between the proposed LargeST and other popular traffic datasets. We next detail the improvements of LargeST over others from three aspects.
Larger Graph Size. LargeST stands out from previous traffic datasets in terms of graph size. For instance, CA includes  $8.4 \times -50.6 \times$  more nodes and  $13.9 \times -729.6 \times$  more edges than its predecessors. Regarding the graph structure, we observe that the sensor graphs released by [30] are extremely sparse, with an average degree of only around 1. This high degree of sparsity significantly limits the effectiveness of graph-based models in capturing the spatial correlations among nodes. Although the situation in the other two data sources is better, they are still not comparable to CA. In short, the increased graph size of LargeST provides a more realistic depiction of the scale of road networks, and offers an excellent testing ground for evaluating the scalability of both current and future traffic forecasting models.
Higher Temporal Coverage. The temporal coverage of existing traffic datasets is often limited, typically spanning no more than 6 months. In contrast, LargeST covers an unprecedented 5 years of traffic data with the same sampling rate as previous datasets, offering several potential benefits. First, it allows for the study of long-term patterns, such as seasonal trends on a weekly or monthly basis. Second, compared to some datasets that date back to 2012, LargeST is more up-to-date and thus better reflects recent traffic conditions. Third, the inclusion of a longer time period provides a larger sample size for model training, which is especially advantageous for deep learning models and becomes even more crucial in the current data-centric AI landscape.
Richer Node Metadata. Existing datasets often lack sufficient metadata for nodes, with some even omitting this important information entirely. This limitation greatly cripples the reliability of the data, since users are left unaware of the precise locations of sensors, impeding their ability to effectively analyze and interpret data and model predictions. Moreover, we argue that traffic forecasting benefits from the provided node features in designing models. For example, node-level information such as the county or highway of a sensor can aid in node clustering [27], which could be helpful in the context of large-scale datasets. In addition, the number of lanes and variability of lane numbers along highways can significantly impact the frequency of accidents [15], which can have a direct effect on traffic flow readings (see Section 4.2.3 for more details).
Table 1: Comparisons between LargeST and other popular datasets. Degree: the average degree of each node. Meta: the number of metadata associated with each node. Data Points: multiplication of nodes and frames. M: million  $(10^{6})$ . B: billion  $(10^{9})$ . The sampling rate for all datasets is at 5 minutes level.
Released Source	Dataset	Nodes	Edges	Degree	Meta	Time Range	Frames	Data Points
Yu et al. [34]	PeMSD7(M)	228	1,664	7.3	6	05/01/2012 – 06/30/2012	12,672	2.89M
PeMSD7(L)	1,026	14,534	14.2	0	05/01/2012 – 06/30/2012	12,672	13.00M
Li et al. [19]	METR-LA	207	1,515	7.3	3	03/01/2012 – 06/27/2012	34,272	7.09M
PEMS-BAY	325	2,369	7.3	3	01/01/2017 – 06/30/2017	52,116	16.94M
Song et al. [30]	PEMS03	358	546	1.5	1	09/01/2018 – 11/30/2018	26,208	9.38M
PEMS04	307	338	1.1	0	01/01/2018 – 02/28/2018	16,992	5.22M
PEMS07	883	865	1.0	0	05/01/2017 – 08/06/2017	28,224	24.92M
PEMS08	170	276	1.6	0	07/01/2016 – 08/31/2016	17,856	3.04M
LargeST (ours)	CA	8,600	201,363	23.4	9	01/01/2017 – 12/31/2021	525,888	4.52B
GLA	3,834	98,703	25.7	9	01/01/2017 – 12/31/2021	525,888	2.02B
GBA	2,352	61,246	26.0	9	01/01/2017 – 12/31/2021	525,888	1.24B
SD	716	17,319	24.2	9	01/01/2017 – 12/31/2021	525,888	0.38B
4 The LargeST Benchmark Dataset
This section formally introduces the proposed LargeST benchmark dataset, which aims to comprehensively evaluate the accuracy, efficiency, and scalability of current and future traffic forecasting models in large-scale scenarios. We start by providing details on how LargeST is collected and organized in Section 4.1. We then conduct a thorough data analysis to gain a deeper understanding of LargeST in Section 4.2, and specify the licenses of data and code in Section 4.3.
4.1 Data Collection and Organization
As shown in Table 1, LargeST comprises four sub-datasets, each characterized by a different number of nodes. These sub-datasets collectively form a hierarchical structure, enabling the evaluation of models at various scales of nodes. We detail the sub-datasets as follows.
We source LargeST from the California Department of Transportation (CalTrans) Performance Measurement System  $^{2}$  (PeMS) [3]. PeMS is an online platform that offers real-time traffic data collected from 18,954 loop detectors (sensors) across the California state highway system. To ensure our dataset represents the overall traffic conditions throughout the entire system, we specifically select sensors labeled as "mainline". We also exclude sensors that lack coordinate information or are extremely far away from other sensors. As a result, we obtain a dataset comprising a total of 8,600 sensors, which we refer to as California (CA).
To undertake a more meticulous analysis of traffic patterns across diverse regions of California, we construct three subsets of CA by selecting three representative areas within CA. The first one is GLA, which contains 3,834 sensors installed in 5 counties of the Greater Los Angeles area: Los Angeles, Orange, Riverside, San Bernardino, and Ventura. The second sub-dataset GBA, includes 2,352 sensors in 11 counties situated in the Greater Bay Area: Alameda, Contra Costa, Marin, Napa, San Benito, San Francisco, San Mateo, Santa Clara, Santa Cruz, Solano, and Sonoma. The smallest sub-dataset SD, comprises 716 sensors only in San Diego county. In addition to the county information, we provide other meta knowledge for each node, including their coordinates, district in PeMS, the highway they are located on, directions of travel, and number of lanes.
Moreover, to build the adjacency matrix of the sensor graph, we follow a common approach used in the field [34, 19, 25], which involves using road network distances. Specifically, we leverage the Open Source Routing Machine  $^{3}$  [24], a high-performance routing engine run on OpenStreetMap  $^{4}$  data, to query the shortest driving distance between sensors based on their coordinates. However,
$^{2}$ https://pems.dot.ca.gov
<sup>3</sup>https://project-osrm.org
<sup>4</sup>https://www.openstreetmap.org/
computing pairwise road network distances can be prohibitively time-consuming when dealing with a large number of nodes. To overcome this, we first compute pairwise geodesic distances between sensors, which is significantly faster than calculating the shortest path between them. Then we limit each node to only query its road network distances to other nodes that are within a 4-kilometer radius. Lastly, we follow Li et al. [19] to normalize the adjacency matrix by setting a small threshold that eliminates weak node connections. In this study, we opt to use a threshold of 0.01 to prune the connections, while ensuring an adequate number of edges are retained. However, the selection of this threshold is adaptable, and users may employ alternative values that align with their specific needs.
Furthermore, LargeST contains five years of traffic flow data (from 2017 to 2021) with a 5-minute interval (same as PeMS), resulting in a total of 525,888 time frames. In this study, we opt not to remove nodes with a high rate of missing traffic flow values. Because we intend to maintain the data in its original state as much as possible, so that users have the discretion to decide whether to fill missing values or not. Moreover, there exist a number of tools/methods to fill the missing values in the time series data. For example, the interpolate() function in the Pandas library is a handy and simple tool to do this.
4.2 Data Analysis
In this part, we conduct a thorough data analysis to unravel the intricate connections between traffic flow and various factors, including space, time, and meta features. We take the traffic data from 2019 as an illustrative example, to reduce the impact of the COVID pandemic.
4.2.1 Regional Disparities
In Figure 2 (a) and (b), for each sub-dataset, we first average the traffic flow across the spatial dimension at each time step, and further partition the data according to weekdays and weekends. In particular, the curve of CA serves as a baseline for the traffic flow level in the state. It is evident that the other three sub-datasets generally demonstrate higher flow values compared to CA. This observation is reasonable since these three sub-datasets correspond to major regions within CA, and other regions in CA typically have lower traffic volume. Comparing the patterns among GLA, GBA, and SD, we observe that they exhibit differences in both flow values and the shape of the curves, particularly on weekdays. These regional disparities can be ascribed to various factors such as differences in economic development levels, area topography, and road planning strategies.
4.2.2 Temporal Dynamics
We continue analyzing Figure 2 (a) and (b) from a temporal standpoint. First, on weekdays, the traffic flow exhibits two prominent peaks, with a morning peak around 8 a.m. and an evening peak typically occurring around 5 p.m. Second, there are notable discrepancies in the daily flow patterns between weekdays and weekends, attributed to the factors of diverse commuting routines, work schedules, and recreational pursuits. These observations underscore the significance of incorporating temporal features like time of day and day of week when designing models.
We next examine potential monthly patterns by visualizing the traffic flow distribution during the morning peak (7 a.m. - 8 a.m.) and evening peak (5 p.m. - 6 p.m.) in CA across 12 months in a year, as depicted in the box plots in Figure 2 (c) and (d), respectively. The results reveal two
Figure 2: Relations between traffic flow and the factors in space and time.
(a) Distribution of Highway Cat.
Figure 3: Relations between traffic flow and two crucial meta features: the highway categories and the number of lanes. The subplots (a) and (c) present the distribution of the features across four datasets. The subplots (b) and (d) show the violin plots of the average traffic flow of the CA dataset categorized by these two features.
(b) Flow vs Highway Cat. on CA
(c) Distribution of #Lanes
(d) Flow vs #Lanes on CA
noteworthy observations. (1) The morning peak displays a larger variance in traffic flow compared to the evening peak. This disparity could be attributed to factors such as flexible working hours in the morning or substantial traffic congestion in the evening. (2) Both peaks undergo a distribution shift throughout the months. This shift is characterized by a gradual increase from January to May, followed by fluctuations from June to September, and subsequently a decline from October to December. This observation aligns with typical human mobility patterns during summer and winter seasons, with increased outdoor activities in summer and reduced activities in winter, and suggests that incorporating the month in year feature could be advantageous for improving model predictions. This also underscores the importance of training and evaluating models using wider temporal ranges, to ensure that they can generalize to such seasonally varying patterns.
4.2.3 Metadata Characteristics
In this part, we focus on analyzing the relationships between traffic flow and two key features: the highway categories and the number of lanes. In Figure 3, we visualize the distribution of two features across four datasets, along with the average traffic flow of sensors in the CA dataset categorized based on the features.
Highway Categories. There are three main kinds of highways in the United States<sup>5</sup>, namely interstate highways, U.S. highways, and state highways. Different types of highways have different road designs, speed limits, and access points, thus impacting traffic patterns. In Figure 3 (a), we can find that the interstate is the most common type while U.S. highway is generally the least common one across four datasets. This aligns with the fact that U.S. highway is an older system and has been gradually replaced by interstate and state routes.
According to Figure 3 (b), which illustrates the average traffic flow of the CA dataset, we observe that interstate highways exhibit the highest average volume, followed by U.S. highways and state highways. This trend aligns with the distinct characteristics of each highway system. Specifically, interstate highways serve as essential transportation arteries that connect major cities, facilitating high volumes of vehicles for daily commutes, long-distance travel, and freight transport. U.S. highways exhibit slightly lower average volumes compared to interstate highways, as they primarily cater to intrastate travel and regional connectivity. State Highways, on the other hand, have the lowest average traffic volume among the three types, due to their localized scope and limited coverage.
Factor of Lanes. Figure 3 (c) presents the distribution of the number of lanes across four datasets, revealing that four lanes are the most prevalent configuration. Moreover, in Figure 3 (d), we show that there exists a positive correlation between the number of lanes on roads and the corresponding traffic volumes. The underlying rationale is as follows. The number of lanes on a road can affect traffic flow by influencing its capacity. Roads with more lanes generally have a higher capacity to accommodate a larger volume of vehicles, allowing for smoother traffic flow and reducing congestion during peak hours [16]. On the other hand, it can also affect traffic flow by influencing drivers' lane-changing behavior. With more lanes available, drivers have more opportunities to switch lanes, which can lead
<sup>5</sup>https://en.wikipedia.org/wiki/Numbered_highways_in_the_United_States
to disruptions in traffic flow. Frequent lane changes can cause merging conflicts, abrupt deceleration or acceleration, and an increased likelihood of accidents [15].
In short, the metadata exhibits strong correlations with traffic flow values, suggesting that they can serve as valuable external knowledge for improving predictive performance.
4.3 LargeST License
The LargeST benchmark dataset is released under a CC BY-NC 4.0 International License: https://creativecommons.org/licenses/by-nc/4.0. Our code implementation is released under the MIT License: https://opensource.org/licenses/MIT. The license of any specific baseline methods used in our codebase should be verified on their official repositories.
Datasets. We conduct experiments on all sub-datasets of LargeST with the setting of predicting the 12-step future based on the 12-step historical data [19, 32]. To facilitate the study of traffic forecasting in a more extended future time period, we aggregate the traffic readings from 5-minute intervals into 15-minute windows, resulting in 96 time steps per day. Moreover, we use one year of traffic data from 2019 in experiments to allow for several less efficient baselines to be compared. We chronologically split the data into train, validation, and test sets, with a ratio of 6:2:2 for all sub-datasets, resulting in sample sizes of 21010, 7003, and 7004, respectively.