SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Baselines. We adopt the following representative traffic forecasting baselines. Historical Last (HL) [21] is a naive method that simply uses the last observation as all the future predictions. LSTM [12] is a temporal-only deep model that does not consider the spatial correlations. Taking advantage of the advancements in GNNs [5, 14], sequential models have been integrated with GNNs to effectively model traffic data. During the period from 2018 to 2020, we specifically select RNN-based methods such as DCRNN [19] and AGCRN [1], TCN-based methods like STGCN [34] and GWNET [32], as well as attention-based methods ASTGCN [10] and STTN [33]. Moreover, we incorporate four representative methods from the years 2021 and 2022, which reflect the recent research directions in the field. STGODE [8] leverages neural ordinary differential equations to effectively model the continuous changes of traffic signals. DSTAGNN [17], DGCRN [18], and  $\mathrm{D}^2$ STGNN [28] specifically consider the dynamic characteristics of correlations among sensors on traffic networks.
Implementation Details. We collect the code of baselines from their respective GitHub repositories, perform necessary cleaning, and integrate them into a single repository to improve the reproducibility and ease of comparison. For the model and training related configurations, we follow the recommended settings provided in their code. Experiments are repeated twice with different seeds on an Intel(R) Xeon(R) Gold 6140 CPU @ 2.30 GHz, 376 GB RAM computing server, equipped with an NVIDIA RTX A6000 GPU with 48 GB memory.
Evaluation Metrics. We conduct a comprehensive comparison of baselines from the following aspects: (1) Performance. We assess the performance of the models using three commonly adopted metrics in forecasting tasks: mean absolute error (MAE), root mean squared error (RMSE), and mean absolute percentage error (MAPE). (2) Efficiency. We consider the efficiency of the models by measuring both the training and inference wall-clock time. We also report the batch size used during training to reflect their ability to handle large-scale datasets. Note that we specify a maximum batch size of 64. If a model is unable to run with this setting, we progressively decrease the batch size until it fully occupies the memory on an A6000 GPU.
5.2 Performance Comparisons
Table 2 presents the test results of MAE, RMSE, and MAPE for specific horizons of 3, 6, and 12, as well as the average values across all predicted horizons. Simple methods such as HL and LSTM
perform worst, since they only consider temporal dependencies and ignore the spatial correlations present in traffic data. The RNN-based method AGCRN and the TCN-based method GWNET generally outperform their predecessors, DCRNN and STGCN, on the SD, GBA, and GLA datasets. Even when compared to recent works like DGCRN and  $\mathrm{D}^2\mathrm{STGNN}$ , their performance remains highly promising. This achievement can be attributed to their utilization of an adaptive adjacency matrix, which serves as a learnable fully-connected graph and significantly enhances model capacity. We also note that ASTGCN and DSTAGNN do not perform well on all datasets, which may be attributed to the model's large number of parameters, leading to potential overfitting issues. Notably, DGCRN and  $\mathrm{D}^2\mathrm{STGNN}$  demonstrate impressive performance on the datasets of SD and GBA, validating the effectiveness of considering the dynamic characteristics of spatial topology. However, their complex model designs prevent them from scaling to larger datasets: GLA and CA. For the performance on the CA dataset, our results show that STGCN and STGODE can outperform GWNET on some of
Table 3: Efficiency comparisons. BS: batch size set during training. Train: training time (in seconds) per epoch. Infer: inference time (in seconds) on the validation set. Total: total training time (in hours). Note that the total training time is also influenced by the spent number of epochs.
Method	SD	GBA	GLA	CA
BS	Train	Infer	Total	BS	Train	Infer	Total	BS	Train	Infer	Total	BS	Train	Infer	Total
LSTM	64	21	6	1	64	115	17	4	64	188	29	6	32	415	61	13
DCRNN	64	867	150	28	64	1,816	319	59	43	2,491	435	81	19	4,845	851	158
AGCRN	64	92	15	3	64	536	83	17	45	1,413	245	46	-	-	-	-
STGCN	64	53	16	2	64	160	54	6	64	268	86	10	64	701	206	25
GWNET	64	97	14	3	64	483	66	15	64	1,028	139	32	44	4,105	548	113
ASTGCN	64	128	19	4	45	1,126	147	35	17	3,060	393	77	-	-	-	-
STTN	64	208	26	6	7	1,758	197	50	-	-	-	-	-	-	-	-
STGODE	64	188	26	6	49	710	103	23	30	1,305	192	42	13	4,212	659	135
DSTAGNN	64	240	23	7	27	1,959	171	53	10	5,241	467	120	-	-	-	-
DGCRN	64	430	76	14	12	4,461	605	138	-	-	-	-	-	-	-	-
D²STGNN	45	563	69	14	4	5,885	796	148	-	-	-	-	-	-	-	-
the metrics. However, they require 2 to  $10 \times$  more parameters compared to GWNET. In summary, we gain two key insights from the table: (1) Competitive methods introduced 3-4 years ago, namely GWNET and AGCRN, continue to perform well across evaluated datasets. Considering that they have been overlooked in some prior works, we believe that their significance should be acknowledged in future studies. (2) We observe that baselines span a wide range of parameter scales. To ensure fairness, evaluating models with comparable parameters is advisable in future research, similar to practices in computer vision and natural language processing.
5.3 Efficiency Comparisons
We summarize the efficiency comparisons in Table 3 with the following observations. LSTM is the fastest method due to its simple model architecture. Apart from this, TCN-based approaches like STGCN and GWNET are generally faster than all other baselines, thanks to the parallel computation of temporal convolution operations. They also demonstrate good scalability on large-scale datasets, as evidenced by the large batch size settings for the CA dataset. Among the RNN-based approaches, DCRNN and AGCRN are slower than TCN-based methods, and AGCRN exhibits significantly better speed than DCRNN due to its utilization of an MLP-like decoder instead of a recurrent decoder. On the other hand, DGCRN and  $\mathrm{D}^2\mathrm{STGNN}$ , despite their strong performance, suffer from long training and inference times. This is due to their complex model architectures that generate a substantial number of intermediate hidden states and thus require significant memory storage. Consequently, a small batch size must be utilized, leading to prolonged training duration. Moreover, the CA dataset poses significant challenges for existing traffic forecasting models, as only half of the selected baselines are capable of running on it. This outcome underscores the importance of developing scalable traffic forecasting models in future research.
6 Future Opportunities & Limitations
To facilitate accurate, efficient, and scalable traffic forecasting research, we introduce LargeST as a new benchmark dataset. It encompasses a total of 8,600 sensors, with each sensor containing 5 years of data and comprehensive metadata. According to the thorough data analysis and extensive experiment results, we highlight the following opportunities in future research.
- The utilization of spatial, temporal, and metadata features. Based on our data analysis, we have discovered positive correlations between traffic flow readings and regional distribution, temporal factors such as day of week and month in year, as well as metadata information like highway categories and lanes number. Future studies may consider incorporating this knowledge to enhance the accuracy and interpretability of traffic forecasting models.
- A valuable testbed for the challenges of temporal distribution shifts. Encompassing a comprehensive five-year temporal period, including the years 2020 and 2021 characterized by the global COVID-19 pandemic, our dataset offers a unique lens into temporal distribution shifts or out-of-distribution challenges. For example, researchers exploring the effects of extraordinary events on forecasting models can leverage our dataset as a testing ground to develop strategies for handling abrupt distribution shifts.
- The development of simple yet effective methods. After analyzing Tables 2 and 3, it becomes apparent that while the proposed methods demonstrate increasing accuracy in recent years, their models also become increasingly complex, which has a significant impact on their efficiency and scalability when applied to larger sensor networks. Therefore, there is a critical need to develop simple yet effective methods in traffic forecasting, to enable their practical implementation and deployment in real-world applications.
- The development of foundation forecasting models. Recently, developing foundation models has raised a surge of interest in a variety of domains, such as ChatGPT in natural language processing and Segment Anything in computer vision. With billions of curated data points, our dataset may serve as an invaluable resource for training foundation models in the fields of traffic forecasting or time series forecasting.
Our work offers a new traffic forecasting benchmark to facilitate research in the field, but it has limitations in terms of the dataset generalizability, as data analysis and all experiments were conducted in the areas of California. Another limitation of the LargeST dataset is associated with inaccuracies and missing data in the sensor readings. These issues can arise from factors such as signal interruptions and other unforeseen circumstances.
This work by the authors is partly supported by the Advanced Research and Technology Innovation Centre (ARTIC), the National University of Singapore under Grant (project number: A-8000969-00-00). This work is also supported by Guangzhou Municipal Science and Technology Project 2023A03J0011, and the National Natural Science Foundation of China (No. 62372402).
[1] Lei Bai, Lina Yao, Can Li, Xianzhi Wang, and Can Wang. Adaptive graph convolutional recurrent network for traffic forecasting. In Proceedings of Advances in Neural Information Processing Systems, pages 17804-17815, 2020.
[2] Defu Cao, Yujing Wang, Juanyong Duan, Ce Zhang, Xia Zhu, Congrui Huang, Yunhai Tong, Bixiong Xu, Jing Bai, Jie Tong, et al. Spectral temporal graph neural network for multivariate time-series forecasting. In Proceedings of Advances in Neural Information Processing Systems, pages 17766-17778, 2020.
[3] Chao Chen, Karl Petty, Alexander Skabardonis, Pravin Varaiya, and Zhanfeng Jia. Freeway performance measurement system: mining loop detector data. Transportation research record, pages 96-102, 2001.
[4] Jeongwhan Choi, Hwangyong Choi, Jeehyun Hwang, and Noseong Park. Graph neural controlled differential equations for traffic forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 6367-6374, 2022.
[5] Michael Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Proceedings of Advances in Neural Information Processing Systems, pages 3837-3845, 2016.
A Dataset Documentation
We organize the dataset documentation based on the template of datasheets for datasets [9].
This dataset is created for comprehensively evaluating the accuracy, efficiency, and scalability of current and future traffic forecasting models in large-scale scenarios. Current public benchmarks in traffic forecasting may not be applicable to practical scenarios due to the following limitations. First, the limited sizes of existing datasets may not reflect the real-world scale of traffic networks. Second, the temporal coverage of these datasets is typically short, posing hurdles in studying long-term patterns and acquiring sufficient samples for training deep models. Third, these datasets often lack adequate metadata for sensors, which compromises the reliability and interpretability of the data.
This dataset was created by Xu Liu, Yutong Xia, Yuxuan Liang, Junfeng Hu, Yiwei Wang, Lei Bai, Chao Huang, Zhenguang Liu, Bryan Hooi, and Roger Zimmermann. The authors are researchers affiliated with the National University of Singapore, Hong Kong University of Science and Technology (Guangzhou), Shanghai AI Laboratory, University of Hong Kong, and Zhejiang University.
There are three types of data in our dataset. (1) The traffic flow readings obtained from loop detectors (sensors) installed in California state highway system. (2) The metadata of sensors. (3) An adjacency matrix that describes the graph topology.
Our dataset has a total number of 8,600 sensors with a 5-year time coverage and includes 9 features for each sensor. It also possesses a sensor graph with 8,600 nodes and 201,363 edges.
Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? If the dataset is a sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic coverage)? If so, please describe how this representativeness was validated/verified.
Our dataset is a sample of instances from a larger set in both spatial and temporal aspects. In terms of spatial coverage, the larger set includes all operational sensors in the California Department of Transportation Performance Measurement System (PeMS)  $^{6}$ . As for the temporal aspect, the larger set consists of all traffic readings since the creation of the PeMS system. We consider our sample to be the representative of the larger set, as we specifically select sensors labeled as "mainline", which
<sup>6</sup>https://pems.dot.ca.gov
provides the overall traffic conditions throughout the entire system. Also, we include 5 years of recent data to ensure sufficient time coverage.
The traffic readings are just numerical values. The metadata of sensors consists of their coordinates, county, district in PeMS, the highway they are located on, directions of travel, and the number of lanes. The adjacency matrix records the connections between sensors.
No, traffic forecasting can be considered as a self-supervised learning task.
Is any information missing from individual instances? If so, please provide a description, explaining why this information is missing (e.g., because it was unavailable).
Yes, there are additional meta features associated with the sensors, such as the cities in which they are located. However, due to a high rate of missing data for these features, we have opted to ignore them in our analysis.
Yes, we use an adjacency matrix to denote the connections between sensors.
Yes, we chronologically split the data into train, validation, and test sets, with a ratio of 6:2:2.
Yes, the sensor readings are never perfectly accurate or sometimes missing due to unexpected factors, such as signal interruption.
Yes, it is self-contained.
Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals' non-public communications)? If so, please provide a description.
No, all our data are from a publicly available data source, i.e., PeMS.
No, all our data are numerical.
How was the data associated with each instance acquired? Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), or indirectly inferred/derived from other data (e.g., part-of-speech tags, model-based guesses for age or language)?
We source the data from the California Department of Transportation (CalTrans) Performance Measurement System (PeMS). The data are directly observable.
What mechanisms or procedures were used to collect the data (e.g., hardware apparatuses or sensors, manual human curation, software programs, software APIs)? How were these mechanisms or procedures validated?
We initiate the dataset construction process by downloading a sensor list from the PeMS official website. This comprehensive list encompasses a majority of sensor meta-information, excluding the coordinates (longitude and latitude). Subsequently, we utilize the sensor IDs from this list to access both the coordinate data and historical observations of specific sensors via the website's APIs. Once we collect and store all the necessary sensor information, we engage in a vital data cleaning process, elaborated in Section A.4. The procedure for forming the adjacency matrix of the dataset is outlined in the main paper. This is basically how the CA dataset is built. We then construct the other three datasets, namely GLA, GBA and SD, through the selection of sensors positioned within these specific geographic areas.
The sampling strategy is deterministic.
Our code collects publicly available data, which is free.
The data are collected in 2023. This timeframe matches the creation timeframe of the data.
Were any ethical review processes conducted (e.g., by an institutional review board)?
No, such processes are unnecessary in our case.
Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? If so, please provide a description.
Yes, to ensure our dataset represents the overall traffic conditions throughout the entire system, we specifically select sensors labeled as "mainline". We also exclude sensors that lack coordinate information or are extremely far away from other sensors.
Was the "raw" data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)? If so, please provide a link or other access point to the "raw" data.
The raw data are available in PeMS. The link is: https://pems.dot.ca.gov.
Is the software that was used to preprocess/clean/label the data available? If so, please provide a link or other access point.
The dataset is used in this paper for the traffic forecasting task.
No, but we may create one in the future.
Traffic data imputation.
We believe that our dataset will not encounter usage limit.
No, users could use our dataset in any task as long as it does not violate laws.
No, it will always be held on GitHub.
How will the dataset will be distributed (e.g., tarball on website, API, GitHub)? Does the dataset have a digital object identifier (DOI)?
The instructions for building LargeST are available at: https://github.com/liuxu77/LargeST. The dataset does not have a digital object identifier currently.
On June 14, 2023.
Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? If so, please describe this license and/or ToU, and provide a link or other access point to.
Our benchmark dataset is released under a CC BY-NC 4.0 International License: https://creativecommons.org/licenses/by-nc/4.0. Our code implementation is released under the MIT License: https://opensource.org/licenses/MIT.
Yes, for commercial use, please check the website: https://pems.dot.ca.gov.
The authors of the paper.
Please contact this email address: liuxu12@u.nus.edu
Users can use GitHub to report issues or bugs.
Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)? If so, please describe how often, by whom, and how updates will be communicated to dataset consumers (e.g., mailing list, GitHub)?
Yes, the authors will actively update the code and data on GitHub.
If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were the individuals in question told that their data would be retained for a fixed period of time and then deleted)? If so, please describe these limits and explain how they will be enforced.
The dataset does not relate to people.
Will older versions of the dataset continue to be supported/hosted/maintained? If so, please describe how. If not, please describe how its obsolescence will be communicated to dataset consumers.
Yes, we will provide the information on GitHub.
If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so? If so, please provide a description. Will these contributions be validated/verified? If so, please describe how. If not, why not? Is there a process for communicating/distributing these contributions to dataset consumers? If so, please provide a description.
Yes, we welcome users to submit pull requests on GitHub, and we will actively validate the requests.
B More Dataset Information
This part endeavors to provide comprehensive details of the proposed LargeST benchmark dataset. Table 4 lists the main statistics of the sub-datasets in LargeST. The traffic flow data are provided in .h5 format, and the adjacency matrices are provided in .npy format. Table 5 shows the node metadata available in our dataset. The metadata file is provided in .csv format. Table 6 presents an overview of the statistical properties of two key features, namely highway categories and the number of lanes, which have been analyzed in the main paper.
Table 4: Dataset statistics. Degree: average node degree. Density: graph density.
Dataset	Nodes	Edges	Degree	Density	Time Range	Sampling Rate	Time Frames
CA	8,600	201,363	23.4	0.0027	01/01/2017 – 12/31/2021	5 minutes	525,888
GLA	3,834	98,703	25.7	0.0067	01/01/2017 – 12/31/2021	5 minutes	525,888
GBA	2,352	61,246	26.0	0.0111	01/01/2017 – 12/31/2021	5 minutes	525,888
SD	716	17,319	24.2	0.0338	01/01/2017 – 12/31/2021	5 minutes	525,888
C More Experimental Settings
We incorporate a total of 12 representative baselines in this study and comprehensively evaluate them on the LargeST benchmark dataset. In our GitHub repository, for each of the baseline, we provide the concrete model and training configurations in a .sh file. Users can directly execute this file to reproduce the reported experimental results. To ensure a fair efficiency comparison, we run all baseline models on the same GPU and measure their training time per epoch, inference time on the validation set, and total training hours.
Moreover, our codebase follows a modular design, allowing users to seamlessly add their custom models by following the established structure and guidelines. This modular approach enhances the codebase's extensibility and adaptability, and encourages experimentation with different model architectures and techniques. In the future, we will keep monitoring new research advancements in the field and actively summarize them in the repository.
Table 5: Metadata description.
Attribute	Description	Possible Range of Values
ID	The identifier of a sensor in PeMS	A number with 6 to 9 digits
Lat	The latitude of a sensor	Real number
Lng	The longitude of a sensor	Real number
District	The district of a sensor in PeMS	3,4,5,6,7,8,10,11,12Sacramento, Yolo, El Dorado, Placer, Nevada,Sutter, Contra Costa, Napa, Solano, SantaCruz, Santa Clara, Alameda, San Benito,Marin, Sonoma, San Francisco, San Mateo,Monterey, Santa Barbara, San Luis Obispo,Kern, Kings, Fresno, Madera, Tulare, LosAngeles, Ventura, San Bernardino, Riverside, San Joaquin, Merced, Stanislaus, San Diego,OrangeI5, US50, SR51, SR65, I80, SR99, SR4,SR12, SR17, SR24, SR25, SR37, SR85,SR87, SR92, US101, SR237, SR238, SR242,I280, I580, I680, I780, I880, I980, SR68,SR41, SR58, SR168, SR180, SR198, SR2,I10, SR14, SR23, SR47, SR57, SR60, SR71,SR91, I105, I110, SR118, SR134, SR170,I210, I405, I605, I710, I15, I215, SR120,SR132, SR219, I8, SR11, SR52, SR54, SR56,SR78, SR94, SR125, SR163, I805, I905,SR22, SR55, SR73, SR133, SR142, SR241,SR261
County	The county of a sensor in California
Fwy	The highway where a sensor is located
Lane	The number of lanes where a sensor is located	1,2,3,4,5,6,7,8
Type	The type of a sensor	Mainline
Direction	The direction of the highway	N, S, E, W
Table 6: Statistic of two meta features.
Dataset	Highway Categories	Number of Lanes
Interstate	U.S.	State	1 to 2	3 to 4	>=5
CA	4,322	1,234	3,044	1,347	5,679	1,574
GLA	2,316	184	1,334	280	2,769	785
GBA	1,108	731	513	349	1,459	544
SD	516	-	200	73	449	194