SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Abhinav Nippani‡, Dongyue Li†, Haotian Ju, Haris N. Koutsopoulos, Hongyang R. Zhang Northeastern University, Boston
{nippani.a, li.dongyu, ju.h, h.koutsopoulos, ho.zhang}@northeastern.edu
We consider the problem of traffic accident analysis on a road network based on road network connections and traffic volume. Previous works have designed various deep-learning methods using historical records to predict traffic accident occurrences. However, there is a lack of consensus on how accurate existing methods are, and a fundamental issue is the lack of public accident datasets for comprehensive evaluations. This paper constructs a large-scale, unified dataset of traffic accident records from official reports of various states in the US, totaling 9 million records, accompanied by road networks and traffic volume reports. Using this new dataset, we evaluate existing deep-learning methods for predicting the occurrence of accidents on road networks. Our main finding is that graph neural networks such as GraphSAGE can accurately predict the number of accidents on roads with less than  $22\%$  mean absolute error (relative to the actual count) and whether an accident will occur or not with over  $87\%$  AUROC, averaged over states. We achieve these results by using multitask learning to account for cross-state variabilities (e.g., availability of accident labels) and transfer learning to combine traffic volume with accident prediction. Ablation studies highlight the importance of road graph-structural features, amongst other features. Lastly, we discuss the implications of the analysis and develop a package for easily using our new dataset.
Graph neural networks are widely used tools for extracting structural relationships from data. Examples include friendships and interactions on social networks [34, 10], 3D protein-protein interactions [61, 45], and road connections on traffic networks [39]. Motivated by the widespread use of graph neural networks, large-scale graph databases and benchmarks have received significant interest in recent studies [26, 14, 25]. Existing architecture designs can be abstracted in the mathematical framework of message-passing neural networks [63]. In practice, the empirical performance of different network designs varies across domains [12]. To facilitate the discussion, this paper examines graph neural networks for the important problem of traffic accident risk modeling: Given historical accident accords as edge labels on a road network, how well can we learn to predict the number of accident occurrences on roads (e.g., over the next month) using graph-structural and related features?
The importance of modeling traffic accident risks is well-recognized as many counties [46, 3, 5, 43] propose vision zero plans to eliminate motor vehicle crashes. According to CDC [56], the economic cost of crashes amounts to \(430 Billion in 2020 [9]. Developing better modeling tools can identify the underlying risk of accidents at a certain location [23], thus informing policy intervention [2].
Many studies have analyzed the effect of road features for predicting accident occurrences, such as traffic flow, road network geometry, and rain [49, 15, 18]. Caliendo et al. [6] conduct a regression
$^\ddagger$ First two authors contributed equally.
(a1) MA, Accidents
(a2) MA, Traffic Volume
(b1) NV, Accidents
Figure 1: We note there is a clear association between accident occurrences and traffic volume. Combining accident and annual average daily traffic reports using transfer learning techniques can improve accident prediction by  $4.6\%$ . Further, road network structural features across states are most predictive of accident occurrences. We capture cross-state variability using multitask learning by combining the labels from all states. This allows information transfer from states with rich data to states with fewer labels. We find that this outperforms learning from individual state data by  $4.7\%$ .
(b2) NV, Traffic Volume
analysis to quantify the safety effects of the annual average daily traffic (AADT) and rain (among others) using observed crash data in Italy. Ihueze and Onwurah [29] incorporate human mobility factors into road accident modeling with autoregressive models. These studies focus on simple regression models. The use of deep learning for traffic accident prediction has recently been examined [41, 52, 73]. Yuan et al. [70] develop a heterogeneous convolutional LSTM framework to predict traffic accidents based on data collected from Iowa state. Moosavi et al. [44] collect over 2 million accident records and develop feedforward networks to predict accident labels on road networks. Despite significant recent interests and strong societal importance, it remains unclear how accurately existing deep learning methods can be used to predict accident occurrences on road networks.
A critical challenge in addressing this question is a lack of (large-scale) traffic accident datasets. There is a large body of work focusing on traffic forecasting (see, e.g., Li et al. [39], Wang et al. [58], and Jiang and Luo [31]), and the datasets therein are usually not annotated with accident records. Moosavi et al. [44] construct a repository including over 2 million accident records collected from map APIs across US states. By contrast, we construct a new dataset with over 9 million traffic accident records spanning eight states, the longest spanning twenty years. We extract these records from official reports of the Department of Transportation from each state, each comes with the latitude, longitude, and time of day of occurrence. This is a nontrivial task as different states publish their data under different formats and APIs, and our dataset unifies them all into the same format. We then collect road networks, road-level AADT, and weather reports aligned with the accident labels in our dataset.
Based on this new dataset, we evaluate existing neural network models in terms of their performance in predicting accident occurrences. Our major finding is that using road structural features and traffic volume reports, existing graph neural networks such as GraphSAGE [20] can predict the accident counts with  $22\%$  mean absolute error (relative to actual counts) and whether an accident occurs on the road with over  $87\%$  AUROC, averaged over eight states. We achieve this result by developing multitask and transfer learning techniques on top of the graph neural network, inspired by recent developments in this space [32, 35, 36]. Interestingly, we notice strong cross-sectional trends regarding graph structural patterns across states, as illustrated in Figure 1. As a remark, we clarify that the results should be interpreted as showing that simple graph neural networks achieve comparable performance to (if not outperforming) their more sophisticated counterparts. Part of the reason is that our dataset is sparsely labeled, with a labeling rate of less than  $0.3\%$  for all states (cf. Table 2), whereas complex models employ a few times more parameters, making them more challenging to fit. One limitation of our analysis is that the road networks, which we extract from OpenStreetMap [19, 4] make simplifying assumptions without considering traffic flows in sequences of connected road segments with typical routes and multi-lanes. We believe our methodology would extend to this case, e.g., by combining our accident labels with satellite image-based construction of road networks [23].
To summarize, this paper makes three contributions to traffic accident analysis by learning road network structures. First, we construct a unified traffic accident dataset extracted from official reports of eight states, totaling 9 million records, the largest dataset of this kind to our knowledge. Second, we
find that by using road network structures and traffic flow reports, graph neural networks can accurately predict accident labels, suggesting the validity of our approach. Third, we discuss the implications of our analysis and develop a package for easily reusing our new datasets and codes. Our datasets and experiment codes can be found at https://github.com/VirtuosoResearch/ML4RoadSafety.
This section presents our approach to collecting and modeling accident data. First, we introduce the problem setup. Second, we describe the collection of a new dataset. Lastly, we present our modeling approach based on graph neural networks, multitask, and transfer learning.
We study the problem of predicting accidents on a road network, viewed as a directed graph  $G = (V, E)$ , where  $V$  denotes a set of road intersections and  $E$  denotes a set of roads that connect one intersection to another intersection. Each node  $v \in V$  has a list of node features, including the number of incoming and outgoing edges, its betweenness centrality, weather information of the corresponding district, etc. An edge  $e \in E$  has features such as the road length, residential or highway category, etc.
Each accident is associated with an edge of the road network where the accident happened, as well as the timestamp during which the accident happened. Given accident records up to a certain time period (e.g., month), we consider the prediction of accident records for the remaining time periods (e.g., months). We measure the result as a regression task by predicting the number of accidents per edge and as a classification task by predicting if one (or more) accidents will occur.
2.2 A unified dataset of crashes and road features
Accident data. Carrying out machine learning for accident analysis requires collecting historical accident information and predictive features. There are several widely used traffic network datasets, such as METR-LA and PEMS-Bay [30, 39]. Note that these curated datasets do not contain vehicle crash information. There are also online data sources that provide available records for the US [11]. We note that the dataset is collected from streaming APIs that only provide accident information for certain times of the day (e.g., during rush hours) [44]. Further, there is some discussion that the data involves reporting errors in the start and end time [11]. Thus, to ensure the validity of our analysis, we start by collecting data from the official reports of the Department of Transportation and note that several states publish detailed information online, including the latitude and longitude of an occurrence. However, extracting this information is nontrivial: Different states provide the data under different formats and interfaces (some in PDF files). It is not obvious how to combine all these records in a unified format. Thus, our first task is constructing a unified dataset with these records.
To this end, we collected over 9 million records spanning eight states in the US. We provide the basic statistics of this dataset in Table 1. We report our statistics at the state level, including the reported number of accidents per million vehicle millage traveled per year, the number of monthly accidents per state, etc. However, it is possible to carry out this analysis at the county level: Some counties, such as New York City and Los Angeles, report traffic accident information within the county (see links in Table 6, Appendix A). For other details of the collection process, see Appendix A.
Annual average daily traffic (AADT) reports. Besides, we explore traffic flow, which is related to crash frequency. The AADT measures the number of vehicles traveling on a particular road. This feature has been known to affect prediction in classical works that apply regression analysis on crash data [49, 15, 18, 6]. However, collecting this data is nontrivial and has not been done in prior works. We collect official reports of road-level AADT from the Department of Transportation, which publishes this information under the name of each street; We map the street names to the edges by extracting a coordinate (using Google Map API) and then aligning it to our graph.
Road features. For each state, we generate its road network based on OpenStreetMap [19, 4]. Note that this protocol has been widely used in prior traffic forecasting studies (e.g., Li et al. [38], Geng et al. [16], and He et al. [22]). In Table 1, we can see that these road networks are very sparse. Besides the graph, we have also collected many other features to help with prediction. For each road, we have its road category and length information. There are 24 categories in total, such as one-way, highway,
Table 1: Below are the statistics of collected traffic accidents and features in eight states. We calculate crash rate as the number of accidents per vehicle millage traveled (VMT) per year using reported millage values from the corresponding state. We use  $d_{avg}$  and  $d_{max}$  to denote the average and maximum degree of a graph, respectively. The volume percentage refers to the fraction of roads for which traffic volume reports are available.
Start	End	Crash Rate	# Nodes	# Roads	# Accidents/Month
Delaware	2009	2022	3.27	49,023	116,196	26,725
Iowa	2013	2022	4.92	253,623	707,072	49,495
Illinois	2012	2021	36.7	627,661	1,647,614	230,666
Massachusetts	2002	2022	24.48	285,942	706,402	70,640
Maryland	2015	2022	11.44	250,565	580,526	87,079
Minnesota	2015	2023	5.39	383,086	979,259	48,963
Montana	2016	2020	1.69	145,525	351,516	17,576
Nevada	2016	2020	5.42	121,392	292,674	35,121
\(d_{avg}\)	\(d_{max}\)	Centrality (10-3)	Avg Length (m)	Volume (%)
Delaware	2.4	6	5.7	213	3.14
Iowa	2.8	7	1.4	532	-
Illinois	2.6	8	0.8	307	-
Massachusetts	2.5	8	0.9	188	1.34
Maryland	2.3	8	1.0	211	1.76
Minnesota	2.5	8	0.6	474	-
Montana	2.4	7	0.02	859	-
Nevada	2.4	6	0.4	280	1.38
residential, and so on (cf. Appendix A). We encode the physical length in meters as a real-valued feature. For each intersection, we compute its in-degrees, out-degree, and betweenness centrality (which measures the ratio of shortest paths between all node pairs that contain a node), apart from its latitude and longitude. These are the static features. Next, we collect historical (daily) weather information for each node, which is temporal in nature. These include the maximum, minimum, and average temperature, the total precipitation of rainfall and snowfall, the average wind speed, and the sea level air pressure. We align each node to the nearest meteorological station to the node.
2.3 Graph neural network modeling for accident prediction
Graph neural networks (GNN). The basic unit for our predictive analysis is graph neural networks: Given a graph  $G = (V, E)$ , along with node-level and edge-level features, a graph neural network applies a neighborhood aggregation mechanism through the edges  $E$ . Let  $l$  be the number of layers. A GNN recursively computes the representations of a node for  $l$  layers by aggregating the representations from its neighbors. Let  $x_{i}^{(k)}$  denote the node feature at node  $i$  in layer  $k$  and  $v_{i,j}$  denote the features of edge  $(i,j)$ . The  $k$ -th layer of a GNN aggregates note  $i$ 's neighborhood embeddings to output:
$$
x _ {i} ^ {(k + 1)} = \phi \left(x _ {i} ^ {(k)}, h \left(\left\{\psi \left(x _ {i} ^ {(k)}, x _ {j} ^ {(k)}, v _ {i, j}\right): j \in N (i) \right\}\right)\right), \text {f o r a n y} i \in V, \tag {1}
$$
where  $h$  denotes an aggregation function (e.g., element-wise sum, mean, and max) and  $N(i)$  is a set of nodes adjacent to  $i$ ,  $\phi$  and  $\psi$  denote neural networks with a nonlinear map such as ReLU.
Cross-state analysis. Next, we develop multitask learning (MTL) models to capture cross-state variability. We note that some states, such as Massachusetts, have way more accident records available than other states, such as Montana. Thus, combining states with more labels can help predict trends for states with fewer labels. However, to ensure that this pooling strategy works, we also require some structural similarities in the feature representations of different states [60].
Interestingly, we observe cross-sectional trends shared across states. In Figure 2, we visualize the number of accidents across four states, including Delaware, Iowa, Illinois, and Massachusetts. In the top panel, we find that the number of accidents gradually increases until 2019 but dips in 2020 due to lockdowns during the pandemic. Nevertheless, after 2020, the accident count increased back.
Multitask learning [8] encodes such inductive bias by using a shared encoder for all tasks. Let  $f(\cdot)$  denote the encoder, e.g., a GNN. For each state, we design a separate prediction layer to map the feature vector to an output. Denote these as  $h_1(\cdot), h_2(\cdot), \ldots, h_k(\cdot)$ . To train these layers, we combine
(a) Delaware
(e1) MA, Winter
Figure 2: 2a-2d: Showing the evolution of annual accident counts across states. There is a sharp drop in 2020 due to the pandemic. 2e1-2f2: Seasonal pattern of accidents, where more accidents occur during winter compared to spring.
(b) Iowa
(e2) MA, Spring
(c) Illinois
(f1) IA & IL, Winter
(d) Massachusetts
(f2) IA & IL, Spring
the data from all states and train a multitask model that yields predictions simultaneously for all states. In particular, we minimize the average loss over the combined dataset of all tasks.
Combining road network geometry and traffic flow. Lastly, we develop a transfer learning (TL) technique to combine traffic flow information, which has been shown to relate to the crash frequency in regression models [49], with network features. We use annual traffic volume reports to support accident prediction by adding another labeling task to our model. In this task, we use the traffic volume information as edge labels beside the traffic records. Given traffic volume up to time  $t$ , we aim to predict traffic volume from  $t + 1$  onwards. This is a regression task at an annual level: Given an edge and a year, the task is to predict the average traffic volume on the road. Importantly, we exclude the traffic volume feature for this prediction. Then, we combine this task together with accident prediction in a multitask learning model. We can view accident prediction as the primary task  $T_{1}$  and volume prediction as an auxiliary task  $T_{2}$ . We jointly train a shared encoder  $f(\cdot)$  and two separate prediction layers  $h_{1}(\cdot)$  and  $h_{2}(\cdot)$  on the averaged loss of both tasks.
We plot the distribution of accidents relative to the corresponding daily traffic volume in Figure 3. For volume, we gather the average daily traffic records for each road and aggregate the number of accidents at various intervals. The result indicates a positive correlation between higher traffic volume and increased accidents. Consequently, one might expect a positive transfer from the volume prediction task to the accident prediction task when they are trained jointly in a shared model [35].
Figure 3: Distribution of accidents by daily traffic volume.
This section evaluates various graph learning methods to predict accident labels based on our new dataset. We focus on three questions. How effective are existing graph learning methods in capturing the patterns of accidents? Does multitask learning capture cross-sectional trends across states in the prediction result? Will traffic volume information help with accident prediction? We provide positive results for the three questions. Additionally, we conduct ablation studies to highlight the importance of graph-structural information and positive correlations at the macro-level as captured by multitask and transfer learning techniques. Lastly, we discuss the implications of our experimental findings.
Baselines. Recall that we consider an edge-level link prediction task. We use both embedding methods and graph neural networks as baselines. First, we test multilayer perceptrons (MLP) using the node features, including node degrees, betweenness centrality, and weather. This tests using node features without network structures. Second, we test embedding methods, including Node2Vec [17] and DeepWalk [48], and add a layer to concatenate the node embeddings and features. Third, we evaluate various GNN architectures, including GCN [34], GraphSAGE [20], and GIN [63]. Lastly, We also evaluate spatiotemporal GNNs that model the temporal dependency of the time serial features. These include DCRNN [39], STGCN [68], AGCRN [1], and Graph Wavenet [62]. To enhance the prediction of GNN, we add Node2Vec embeddings as node features.
We consider three methods to conduct multitask and transfer learning: First, we fit multitask GNNs by combining the data of all eight states. We denote this model as MTL. Second, after fitting an MTL model, we fine-tune the feature encoder on an individual state's data to personalize its representations. We denote this model as MTL-FT. Third, we train a model on accident and volume prediction jointly. We denote this model as TL. For all methods above, we use GraphSAGE as the feature encoder.
**Implementations.** For all baselines, we construct the feature representation for each edge by concatenating the two adjacent nodes' encoded representations and the edge-level features. In our implementation, we fix the dimension of node embeddings as 128. We use two-layer MLP and GNN with a hidden dimensionality of 256. We train our models using Adam as the optimizer. We use a learning rate of 0.001 for 100 epochs on all models. These hyper-parameters are tuned with grid search on the validation set for all the models. The number of layers is tuned in a range of  $\{2,3,4\}$ . The hidden dimensionality is tuned in a range of  $\{128,256,512\}$ . The learning rate is tuned in a range of  $\{0.01,0.001,0.0001\}$ . The number of epochs is tuned in a range of  $\{50,100,200\}$ . See Appendix B for more implementation details.
For each state, we evenly split the available period of accidents into training, validation, and test sets. We split the accidents according to time. We use past accident records until a specific year to train the models and evaluate the model's performance on future accidents occurring after that year. We focus our prediction monthly and add up the daily number of occurrences, but note that one can use our datasets to conduct the analysis at a daily or annual level too. We use both regression and classification metrics to evaluate the results. For regression, we measure the mean absolute error (MAE) between the predicted number of accidents and the actual number of occurrences on a particular road. For classification, we measure AUROC scores.
We summarize our experimental results in Table 2. We highlight three conceptual takeaways below, which we believe will also apply more broadly beyond the specific setting of our experiment.