SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(1) Graph neural networks can accurately predict accident labels. We find that using graph neural networks can predict accident counts with 0.3 mean absolute error, which is  $22\%$  relative to the absolute accident count on average over eight states. For classifying whether an accident occurs or not, GNNs can achieve  $87\%$  AUROC score on average.
Among the GNNs, we observe that the comparisons between GraphSAGE and other spatiotemporal GNNs are generally mixed, with none of them dominating each other. While GraphSAGE has fewer parameters, its performance is still comparable with (if not outperforming) alternative models with two times or more parameters. One explanation is that spatiotemporal GNNs have more trainable parameters than GraphSAGE. Therefore, they need more training labels to fit the model. On the other hand, the labeling rate of our dataset, i.e., the percentage of edges with a positive label or accident occurrence, is around or below  $0.2\%$ , as shown in Table 2 (under the row of “Positive Rate”).
(2) Multitask learning captures macro-level trends across states. We also find that multitask learning outperforms single-task learning (STL) by relatively  $8.4\%$  in terms of MAE and  $0.9\%$  in terms of AUROC averaged over eight states. This is achieved by first training a model on the combined data of all states and then fine-tuning the MTL model on each individual state data.
(3) Transfer learning with AADT improves test performance. Lastly, we find that combining traffic volume and accident prediction yields a relative improvement of  $7.9\%$  in MAE and  $1.1\%$  in AUROC over STL, averaged over the four states with traffic volume records.
Table 2: We compare the experimental results across eight states using node embedding methods and graph neural networks. We also include multitask and transfer learning results for each state. We report the results obtained under the mean absolute errors (MAE) on the test split. We also report the AUROC score on the test split. To measure standard deviations, we run the same experiment over three different random seeds and report the averaged result.
MAE (↓)
Avg Count	DE
1.23	IA
1.14	IL
1.33	MA
2.27	MD
1.22	MN
1.18	MT
1.16	NV
1.38
MLP	1.4±0.07	0.3±0.02	1.4±0.17	1.0±0.11	0.4±0.02	0.3±0.02	0.4±0.01	0.5±0.01
Node2Vec	1.1±0.18	0.3±0.01	0.7±0.05	1.3±0.51	0.4±0.03	0.4±0.02	0.2±0.03	0.3±0.01
DeepWalk	0.8±0.05	0.3±0.01	0.6±0.03	1.0±0.06	0.4±0.01	0.4±0.01	0.3±0.03	0.3±0.02
GCN	0.6±0.02	0.3±0.02	0.5±0.06	0.7±0.02	0.4±0.04	0.3±0.00	0.2±0.03	0.3±0.02
GraphSAGE	0.3±0.01	0.3±0.01	0.4±0.03	0.8±0.02	0.4±0.01	0.3±0.01	0.2±0.02	0.2±0.01
GIN	0.8±0.02	0.3±0.04	0.4±0.04	1.1±0.02	0.4±0.02	0.3±0.06	0.2±0.08	0.2±0.05
AGCRN	0.3±0.01	0.3±0.01	0.4±0.01	0.7±0.01	0.4±0.01	0.3±0.01	0.2±0.01	0.2±0.04
STGCN	0.2±0.02	0.3±0.00	0.4±0.06	0.8±0.06	0.5±0.01	0.3±0.01	0.2±0.01	0.2±0.01
Graph Wavenet	0.3±0.03	0.3±0.02	0.4±0.00	0.7±0.01	0.3±0.00	0.3±0.01	0.3±0.03	0.2±0.01
DCRNN	0.3±0.01	0.3±0.02	0.4±0.03	0.9±0.06	0.3±0.01	0.4±0.02	0.2±0.02	0.2±0.03
MTL	0.2±0.02	0.2±0.01	0.4±0.02	0.7±0.02	0.3±0.00	0.2±0.01	0.1±0.00	0.2±0.01
MTL-FT	0.2±0.01	0.2±0.00	0.2±0.00	0.7±0.01	0.3±0.00	0.2±0.01	0.1±0.00	0.2±0.00
TL	0.2±0.01	-	-	0.6±0.02	0.3±0.01	-	-	0.2±0.01
AUROC (↑)
Training Size	DE
93,184	IA
187,046	IL
646,739	MA
540,682	MD
283,226	MN
124,435	MT
34,475	NV
73,164
Positive Rate	0.23	0.07	0.14	0.10	0.15	0.05	0.05	0.12
MLP	75.5±0.6	68.7±0.1	71.4±0.3	70.6±0.2	74.1±0.4	76.7±0.2	71.6±0.1	56.2±0.1
Node2Vec	83.5±0.1	83.8±0.1	77.4±1.5	70.7±0.4	83.5±0.1	80.6±0.1	84.9±0.1	91.8±0.1
DeepWalk	83.4±0.3	81.6±0.2	78.6±0.2	69.5±0.2	83.7±0.1	80.5±0.2	85.0±0.1	91.8±0.1
GCN	83.2±0.1	85.4±0.1	84.7±1.0	70.6±0.1	83.2±0.1	84.3±2.2	87.4±0.1	91.9±0.6
GraphSAGE	87.6±0.1	84.8±0.2	87.0±0.2	81.8±0.1	87.6±0.1	83.8±2.8	87.5±0.1	91.6±1.0
GIN	82.6±0.7	83.5±0.1	84.2±1.4	68.9±0.5	82.6±0.7	85.4±1.4	85.4±0.1	91.4±1.0
AGCRN	86.0±0.2	83.9±0.2	86.3±0.3	82.1±0.2	88.5±0.1	81.8±0.7	84.3±0.4	90.7±0.2
STGCN	85.4±0.1	83.5±0.1	85.2±0.4	81.9±0.3	88.7±0.1	81.5±0.2	83.8±0.3	91.5±0.3
Graph Wavenet	85.0±0.2	83.9±0.2	85.8±0.2	81.9±0.5	87.9±0.1	80.3±0.1	83.4±0.2	90.6±0.2
DCRNN	81.2±1.2	81.8±0.1	80.7±0.0	70.5±0.1	84.5±0.3	79.3±0.4	81.9±0.5	90.5±0.7
MTL	87.7±0.1	81.7±0.2	84.4±0.3	79.6±0.1	88.7±0.1	87.9±0.0	88.4±0.2	90.3±0.2
MTL-FT	87.8±0.3	84.9±0.2	87.2±0.2	81.9±0.3	88.1±0.1	87.6±0.3	88.5±0.3	91.8±0.2
TL	87.3±0.2	-	-	82.6±0.2	87.9±0.4	-	-	92.8±0.1
3.3 Ablation studies
Influence of graph-structural features. We conduct a leave-one-out analysis of different categories of features for accident prediction. These include graph-structural features, weather, and traffic volume. We remove one type of feature at a time and compare the performance after leaving out a particular feature. Removing graph-structural features reduces the performance by  $6.9\%$ . On the other hand, removing weather and traffic volume reduces performance by  $2.3\%$  and  $1.2\%$ , respectively. See Table 3 for the results, which justifies graph-structural features are the most significant features.
Positive pairwise transfer across states. Next, we measure the transfer effects between every pair of states. For each state, we view it as a source state and consider how well it would transfer to another target state. To test this, we conduct MTL by combining each state with another state's data. This leads to a total of 28 pairwise MTL models. We show the results in Figure 4 on the right. To help read this table, we subtract each MTL model's performance for a target state from the STL performance of that target state. Thus, a positive value in the table indicates a positive transfer from the source state to the target state. We find that for most pairs, the effect is positive.
Figure 4: Pairwise MTL vs STL.
Transferability from traffic volume to accident prediction. We also measure the transfer effect from volume prediction to accident prediction. For one state, such as the MA, we first fit a model on the volume prediction task. Then, we fine-tune this model on the accident prediction task using
Table 3: We evaluate the influence of graph-structural structure, weather, and traffic volume information. We report the test AUROC of accident classification by removing each feature type in the network. We also include the transfer results from traffic volume to accident prediction.
DE	MA	MD	NV
Using all features	87.61±0.10	81.80±0.12	87.51±0.02	91.62±0.99
w/o graph structural features	81.25±0.51	79.63±0.18	79.77±0.94	82.56±0.09
w/o weather information	87.27±0.32	80.71±0.26	80.22±0.45	90.38±0.88
w/o road information	82.99±0.54	81.65±0.77	80.63±0.52	84.24±0.41
w/o traffic volume information	87.15±0.49	80.94±0.31	86.17±1.15	91.58±0.99
Combining traffic volume prediction	87.78±0.07	82.18±0.14	87.77±0.24	92.07±0.57
the accident labels while keeping the same network and other features. This fined-tuned model outperforms single-task learning (STL) by  $0.6\%$ , averaging over four states with volume labels.
Sensitivity analysis. Lastly, we study the hyper-parameters used in our experiments. We set the hyper-parameters as follows: the number of layers is 2, the hidden dimensionality is 256, the learning rate is  $1e^{-3}$ , and the number of epochs is 100. We then vary one hyper-parameter at a time and keep the others unchanged. We notice that using the number of layers as 2, hidden dimensionality as 256, and learning rate as  $1e^{-3}$  yields the best results for all baselines. The validation performance stops improving after training up to 100 epochs. Thus, we adopt these settings as the default parameters.
3.4 Interpretations and implications
Our findings provide some evidence to show that the road network structure is highly predictive of traffic accident occurrences. The evaluation of numerous graph learning methods shows that these methods are valid for accident analysis. Our findings about the road structures go beyond existing regression analysis in the transportation literature (e.g., [49, 6, 41]) thanks to our large-scale dataset. Based on the findings, we discuss several implications for when our datasets might be helpful.
Studying the effect of policy interventions: One way to use our dataset is for policy interventions by examining the accident patterns before and after the implementation of this policy. Our modeling approach could be useful in policy making such as the abolition of mandatory vehicle inspection. The model predictions can provide counterfactual comparisons to facilitate such discussions.
Variability at the county level: With our dataset, it is also possible to study road structures across different counties within the same state. This controls for variability with weather conditions, and we believe the comparison can potentially inform network design as well.
Better reporting of volume information: We believe that better reporting of traffic volume information could be very useful for accident analysis. In our dataset, we notice that the percentage of roads where volume reports are available is pretty low (cf. Table 1). For example, we noticed that on some roads the accident numbers are high but the traffic volume is either low or missing. Besides, we also note that it is likely there is under-reporting of accidents (e.g., to reduce insurance costs). Thus, having better reports of traffic flow combined with better modeling can help address such questions.
Graph datasets and benchmarks. Datasets and benchmarks are instrumental to machine learning research. Motivated by developments in machine learning on graphs, several large-scale graph learning benchmarks have been developed [26, 45, 14, 25]. Chiang et al. [10] construct a large-scale graph dataset corresponding to Amazon's product co-purchase information. MoleculeNet [61] introduces a large-scale benchmark for studying molecule graphs and structures. See also recent works by Townshend et al. [55], who study three-dimensional representations of molecules, and Paetzold et al. [47], who provide whole-brain vessel graphs based on whole, segmented murine brain images. TAPE [51] describes a set of five biologically relevant semi-supervised learning tasks spread across different domains of protein biology. TUDataset [45] consists of over 120 datasets of varying graph sizes from various applications.
Table 4: Comparison of our dataset to the work of Huang et al. [27] and Yuan et al. [70].
Huang et al. [27]	Yuan et al. [70]	Our Dataset
Data Source	Based on work by Moosavi et al. (2019), sourced from Microsoft Bing Map Traffic	Department of Transportation's official accident reports	Department of Transportation's official accident reports
Coverage	2.8 million records across 13 states for 2016-2021	Iowa, for 2006-2013	9 million records, across 8 states, for a maximum of 20 years up to 2023
Prediction Tasks	Node-level classification, evaluated on AUROC	Spatial grid-level regression, evaluated under MAE	Edge-level regression/classification, evaluated under MAE/AU-ROC
Features	Road network features	Averaged grid-level features, traffic volume, rainfall features	Traffic volume, weather features, road network features
The open graph benchmark and database [26, 14, 25] involve academic collaboration networks, protein structural graphs, Reddit social networks, and carefully designed data splits to facilitate comparison between methods. Besides supervised learning, recent work has developed benchmarks for graph contrastive learning [75], continual graph learning [72], node outlier detection [42], and anomaly detection [28]. Compared with existing datasets, ours concerns a problem that is of relevance to society, and we provide spatiotemporal patterns across states.
Spatiotemporal graph mining. The importance of spatial and temporal features for time series prediction is well recognized [7, 58, 40]. Convolutional networks are often used to process time series graphs [68, 59]. Besides, multitask and meta-learning is used to tackle spatiotemporal heterogeneity [38, 65, 57]. Zhuang et al. [76] quantify the uncertainty of spatiotemporal GNN. Lastly, the transferability of graph representations has recently been studied by Zhu et al. [74] for airport networks and gene interaction graphs. Our results suggest that road network structures are highly transferable in predicting traffic accidents. The use of contrastive learning on graph-structured data has been extensively explored in recent work [67, 66]. The following studies [50, 71] have formulated contrastive learning methods on spatiotemporal graph data, such as urban flow prediction, crime prediction, and house price prediction. One interesting question is to revisit these techniques within the context of traffic accident prediction for road safety.
Data-driven traffic forecasting. There is a vast literature on traffic network analysis such as traffic forecasting [31]. Public datasets such as METR-LA and PEMS-BAY are widely used in ML research. These are collected from loop detectors by California's Transportation Agencies. Many research studies use taxi data, such as the NYC open dataset and Didi Chuxing traffic information. Recent work has studied constructing road networks from satellite images [22]. Some recent studies of traffic forecasting are driven by ride-hailing platforms [16]. There are also studies on predicting the estimated time of arrival [24], traffic time and distance for a taxi trip [38], and traffic speed [21].
Comparison with traffic accident prediction datasets. Previous works [70, 27] and our work are targeted at large-scale traffic accident prediction. Similar to our work, both works use OpenStreetMaps to construct road network features. Besides, our work and previous works include similar road network features, including the length of a road and the type of road (e.g., highway or residential, one-way).
The difference between our work and previous works is primarily in constructing the accident information. First, TAP [27] used the accident information collected from another work by Moosavi et al. [44], sourced from Microsoft Bing Map Traffic. Their dataset includes a total of 2.8 million accident records, which is for five years (2016-2021). In contrast, our datasets are collected from the Department of Transportation's official accident reports. Our dataset includes 9 million records for a maximum of 20 years (e.g., Massassachusetts, from 2002 to 2023, which also contains more recent data). Second, each accident record contains the (latitude and longitude) of the incident. We map each coordinate to the nearest edge, leading to an edge-level classification/regression problem, whereas TAP maps each record to the nearest node, resulting in a node-level classification setting. Thus, our work can be evaluated under both MAE/AUROC metrics. Another major difference is that we have collected traffic volume and weather features alongside the road network features. In summary, we give a detailed comparison between our dataset and two existing datasets in Table 4.
Table 5: Comparison of our dataset to several existing spatiotemporal datasets.
METR-LA [39]	PEMS-Bay [39]	Taxi NYC [54]	Didi Chuxing [38]	Our Dataset
Targets	Traffic speed and volume	Traffic speed and volume	# Rides in a region	Trip time	Traffic accident
Coverage	6 million for 4 months from Mar. 2012 to Jun. 2012	16 million for 6 months: Jan. 2017 - May 2017	35 million over 5 years: Jan. 2011 - Jun. 2016	61.4 million over six months: May. 2017 - Oct. 2017	9 million in 8 states up to 20 years until 2023
Data Source	Highway loop detectors	CalTrans	Taxi GPS data for NYC	Didi Chuxing Beijing	Department of Transportation
Features	Traffic speed and volume	Traffic speed and volume	Traffic flow, latitude, longitude, distances	Travel distance, # road, lights, turns for each ride	Traffic volume, weather, and road network features
Comparison with spatiotemporal datasets. Next, we compare several spatiotemporal datasets with ours. We note that existing datasets focus on rather different tasks. Li et al. [39], proposes to predict traffic speed and volume collected from highway loop detections. Some other works study to predict traffic information from ride-hailing platforms, such as the ride number in a certain region [54] and the estimated trip time [38]. In contrast, our work aims to predict traffic accidents. In addition, our dataset provides extensive coverage over 8 states. Other spatiotemporal datasets are mostly constructed for a single city. In contrast, our dataset covers a broader range of areas and facilitates the study of cross-sectional trends beyond a single area. For details, see Table 5 below.
Interpretations and future directions. It is worth pointing out that our goal is not to identify exactly where new accidents will occur since this is impossible. Instead, our results can be interpreted as providing suggestions for the risk or hazard of a particular road, whether or not an accident will occur [23]. We can use the predicted accident count as a risk oracle for guiding drivers' behavior [2]. With this in mind, we discuss several promising questions for future work. First, one can consider the accident prediction problem with our dataset and methodology but at different granularities. One can capture variability within a week by predicting at a daily level or even an hourly level [73]. This can be used to compare risks between weekdays vs. weekends and holidays or rush hour vs. evening hours. Second, one can develop novel time series graph learning methods such as autoregressive models [29]. Besides predicting crash frequency, another relevant statistic to consider is crash rate as normalized by vehicle millage. Third, it would be interesting to see if recent developments in road network construction from satellite images can be used to design new predictive features. More broadly, there are ample opportunities to use our datasets to study policy interventions on road safety.
The ML4RoadSafety package. To facilitate further research, we have developed a package to make our datasets easily accessible to researchers. Our package uses the data format as the existing graph learning library, PYTORCH GEOMETRIC, fully compatible with PYTORCH. A user can have access to our dataset with a single line of code, by only specifying the name of a state, such as Massachusetts. The package will automatically download, store, and return a dataset object. Then, our package offers functions to obtain the accident records and network features for a particular month. Besides the datasets, we provide a trainer module to train and evaluate a GNN on our datasets. Using the trainer, the user can easily implement other training techniques, including multitask learning, transfer learning, and contrastive learning. We provide code examples in Appendix B.3.
We collected a large-scale dataset of 9 million traffic accident records across eight states to analyze traffic accident occurrences using road networks, traffic flow, and weather reports. Through extensive experiments, we found that existing graph neural networks can be used to predict accident labels with over  $87\%$  AUROC score. This uses multitask and transfer learning techniques on graphs. Our analysis reveals strong cross-sectional similarities across states regarding road network structures. Ablation studies validate the importance of graph-structural features for achieving the results.
Acknowledgement. Thanks to Hari Balakrishnan for an invited talk at Northeastern that brought the problem [2] to our attention. Thanks to Ravi Sundaram for several discussions. This research is supported in part by Northeastern University's Transforming Interdisciplinary Experiential Research (TIER) 1: Seed Grant/Proof of Concept Program.
[63] K. Xu, W. Hu, J. Leskovec, and S. Jegelka (2019). "How powerful are graph neural networks?" In: ICLR (pages 1, 6).
[64] F. Yang, H. R. Zhang, S. Wu, C. Ré, and W. J. Su (2020). "Precise High-dimensional Asymptotics for Quantifying Heterogeneous Transfers". In: arXiv preprint arXiv:2010.11750 (page 24).
[65] H. Yao, Y. Liu, Y. Wei, X. Tang, and Z. Li (2019). "Learning from multiple cities: A meta-learning approach for spatial-temporal prediction". In: WWW (page 9).
[66] Y. You, T. Chen, Y. Shen, and Z. Wang (2021). "Graph contrastive learning automated". In: International Conference on Machine Learning. PMLR, pp. 12121-12132 (pages 9, 23).
[67] Y. You, T. Chen, Y. Sui, T. Chen, Z. Wang, and Y. Shen (2020). "Graph contrastive learning with augmentations". In: Advances in neural information processing systems (pages 9, 21, 23).
[68] B. Yu, H. Yin, and Z. Zhu (2017). "Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting". In: IJCAI (pages 6, 9, 23).
[69] H. Yuan, H. Yu, S. Gui, and S. Ji (2022). "Explainability in graph neural networks: A taxonomic survey". In: IEEE transactions on pattern analysis and machine intelligence (page 24).
[70] Z. Yuan, X. Zhou, and T. Yang (2018). "Hetero-convlstm: A deep learning approach to traffic accident prediction on heterogeneous spatio-temporal data". In: KDD (pages 2, 9).
[71] Q. Zhang, C. Huang, L. Xia, Z. Wang, Z. Li, and S. Yiu (2023). "Automated Spatio-Temporal Graph Contrastive Learning". In: The Web Conference (pages 9, 24).
[72] X. Zhang, D. Song, and D. Tao (2022). "Cglb: Benchmark tasks for continual graph learning". In: Neural Information Processing Systems Datasets and Benchmarks Track (page 9).
[73] Z. Zhou, Y. Wang, X. Xie, L. Chen, and H. Liu (2020). "RiskOracle: a minute-level citywide traffic accident forecasting framework". In: AAAI (pages 2, 10).
[74] Q. Zhu, C. Yang, Y. Xu, H. Wang, C. Zhang, and J. Han (2021a). "Transfer learning of graph neural networks with ego-graph information maximization". In: NeurIPS (page 9).
[75] Y. Zhu, Y. Xu, Q. Liu, and S. Wu (2021b). "An empirical study of graph contrastive learning". In: NeurIPS 2021 Datasets and Benchmarks track (page 9).
[76] D. Zhuang, S. Wang, H. Koutsopoulos, and J. Zhao (2022). "Uncertainty Quantification of Sparse Travel Demand Prediction with Spatial-Temporal Graph Neural Networks". In: KDD (page 9).
A Data Collection Procedure
In this section, we describe the details of the collection of our dataset. We have provided the implementation in our code repository: https://github.com/VirtuosoResearch/ML4RoadSafety. We have also uploaded the entire dataset to an open repository: https://doi.org/10.7910/DVN/V71K5R. This section serves as the documentation for the collection process. For reference, we have collected the links to public data sources where we extracted our dataset below.
Table 6: Links to the data sources from which we extracted our dataset.
Traffic accident records
Delaware Open Data	https://data.delaware.gov/Transportation/Public-Crash-Data-Map/3rrv-8pfj
Delaware DOT	https://deldot.gov/search/
Iowa DOT	https://icat.iowadot.gov/
Illinois DOT	https://gis-idot.opendra.arcgis.com/search?collection=Dataset&q=Crashes
Mass DOT	https://apps IMPACT.dot.state.ma.us/cdp/home
Maryland Open Data	https://opendata.maryland.gov/Public-Safety/Maryland-Statewide-Vehicle-Crashes/65du-s3qu
MN Crash	https://mncrash.state.mn.us/Pages/AdHocSearch.aspx
Montana DOT	https://www.mdt.mt.gov/publications/datassts/crashdata.aspx
Nevada DOT	https://ndot.maps.arcgis.com/apps/webappviewer/index.html?id=00d23dc547eb4382bef9beabe07eaefd
Road networks
OSMnx Street Network Dataverse	https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/CUWWYJ
OpenStreetMap Road Categories	https://wiki.openstreetmap.org/wiki/Highways#Classification
Google Map API	https://maps.google.com/
Weather reports
Meteostat API	https://meteostat.net/en/
Traffic volume reports
Delaware	https://deldot.gov/search/
Maryland	https://data-maryland.opendra.arcgis.com/datasets/mdot-sha-annual-average-daily-traffic-aadt-segments/explore?location=38.833256%2C-77.269751%2C8.30&showTable=true
Massasuchetts	https://mhd.public.ms2soft.com/tcds/tsearch.asp?loc=Mhd&mod= https://www.mass.gov/lists/massdot-historical-traffic-volume-data
Nevada	https://geohub-ndot Hub.arcgis.com/datasets/trina-stations/explore?location=38.490765%2C-116.969086%2C7.27&showTable=true
A.1 Traffic accident records