SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Bang An
bang-an@uiowa.edu
Xun Zhou *
xun-zhou@uiowa.edu
Yongjian Zhong
yongjian-zhong@uiowa.edu
The problem of urban event ranking aims at predicting the top- $k$  most risky locations of future events such as traffic accidents and crimes. This problem is of fundamental importance to public safety and urban administration especially when limited resources are available. The problem is, however, challenging due to complex and dynamic spatio-temporal correlations between locations, uneven distribution of urban events in space, and the difficulty to correctly rank nearby locations with similar features. Prior works on event forecasting mostly aim at accurately predicting the actual risk score or counts of events for all the locations. Rankings obtained as such usually have low quality due to prediction errors. Learning-to-rank methods directly optimize measures such as Normalized Discounted Cumulative Gain (NDCG), but cannot handle the spatiotemporal autocorrelation existing among locations. In this paper, we bridge the gap by proposing a novel spatial event ranking approach named SpatialRank. SpatialRank features adaptive graph convolution layers that dynamically learn the spatiotemporal dependencies across locations from data. In addition, the model optimizes through surrogates a hybrid NDCG loss with a spatial component to better rank neighboring spatial locations. We design an importance-sampling with a spatial filtering algorithm to effectively evaluate the loss during training. Comprehensive experiments on three real-world datasets demonstrate that SpatialRank can effectively identify the top riskiest locations of crimes and traffic accidents and outperform state-of-the-art methods in terms of NDCG by up to  $12.7\%$ .
Given a risk score defined based on historical urban events (e.g., crimes, traffic accidents) in a study area as well as the socio-environmental attributes (e.g., travel demands, weather, road network) associated with the events, the goal of the urban event ranking problem is to learn a model that can predict the ranking of the top- $k$  riskiest locations of future events.
The urban event ranking problem is a widely observed spatiotemporal prediction problem, which has critical applications in public safety, traffic management, and urban planning. For example, the National Highway Traffic Safety Administration [27], estimated a total of 42,939 deaths in motor vehicle traffic crashes in the United States in 2021. Crime control and property loss cost over $2.5 trillion in the United States in 2017 [25]. Chicago Police Department has utilized criminal intelligence analysis and data science techniques to help command staff determine where best to deploy resources [1]. However, according to the Police Executive Research Forum, there were  $42.7\%$  more resignations among law enforcement but a  $3.9\%$  decrease in hiring new officers in 2021 compared to 2019 [3]. Meanwhile, the Federal Bureau of Investigation confirms that violent crime in 2020 has surged nearly  $30\%$  over 2019 [2]. Therefore, given such growth of crimes and accidents, predicting the riskiest locations of traffic accidents or crimes helps law enforcement stakeholders allocate their limited resources strategically to reduce injuries, deaths, and property losses.
Despite its importance, the urban event ranking problem is technically challenging. First, there exist complex and dynamic spatiotemporal correlations between locations in terms of events and attributes such as traffic conditions and weather changes. Capturing such dynamic relationships is non-trivial. Second, due to the presence of spatial autocorrelation, nearby locations may share very similar socio-environmental attributes. It is thus very challenging to learn a good model that can correctly rank neighboring locations. Finally, urban events are usually sparse in space. Making a prediction with a low average error in count does not ensure a good ranking of the top- $k$  locations.
Prior works on urban event prediction employed either classic machine learning methods [15] [4] [7] [9] [11] [12] [18] or deep learning techniques including recurrent neural networks [30] and convolutional neural networks [26][36]. Recently, Li et al. [20] proposed a cross-region hypergraph structure network to address the data sparsity issues. Besides, Yuan et al. [40] [5] proposed spatial ensembles to address spatial heterogeneity. The objective of the above methods is to make accurate predictions of the risk or count for all locations. However, urban events are often very sparse in space, which might guide the model to avoid predicting accidents in most locations to obtain a low average error. Such prediction does not truly benefit the users such as police officers. The top- $k$  locations derived from such predictions are naturally inaccurate.
Our problem is also relevant to the learning-to-rank problem frequently studied in the field of recommendation systems. State-of-the-art methods in this area typically predict the top- $k$  items most likely to be chosen by users through optimizing ranking-based metrics such as the Normalized Discounted Cumulative Gain (NDCG) [8]. The rank operator of NDCG is non-differentiable, and previous studies have made noticeable progress in approximating NDCG by surrogate functions [32] [28] [34] [29]. In our problem, a straightforward adaptation would be to consider the locations as “items” and each time slot as a “user”. However, NDCG is not a perfect objective function for our problems as it neither measures the local ranking quality nor considers spatial autocorrelation among locations. Instead, the existing approaches assume items are independent. Therefore, directly applying existing NDCG optimization solutions might not be the best solution to our problem.
In this paper, we bridge the gaps in both fields by formulating the urban event ranking problem as a spatial learning-to-rank problem and solving it by directly optimizing a "spatial" version of the NDCG measure. We propose SpatialRank, our deep learning model with three novel designs. To efficiently capture the spatial and temporal dependencies, we design an adaptive graph convolution layer that learns the correlations between locations dynamically from features and historical events patterns; to ensure the model balances both the global ranking quality on all the locations and the local ranking quality on subsets of locations, we propose a hybrid loss function combining both NDCG loss and a novel local NDCG loss; to improve the effectiveness of optimizing NDCG surrogates in the spatiotemporal setting of our problem, we design an importance-based location sampling with spatial filtering algorithm to iteratively adjust the weights of each location considered in the objective function, thereby guiding the model to concentrate on learning for more important locations. We conduct comprehensive experiments on three real-world datasets collected from Chicago and the state of Iowa. The results demonstrate that SpatialRank can substantially outperform baselines and achieve better ranking quality.
- To the best of our knowledge, this is the first paper to formulate urban event forecasting as a location ranking problem and learn a ranking model by optimizing its ranking quality.
- We propose SpatialRank with adaptive graph convolution layers learning from historical event patterns and spatiotemporal features to capture dynamic correlations over time and space.
- We propose a hybrid objective function to leverage the trade-off between global ranking quality and local ranking quality.
- We propose a ranking-based importance sampling algorithm to adaptively adjust the weights of different locations considered in the objective function of the prediction results from the last training epoch to help the model focus on important locations.
2.1 Formulation of the event ranking problem
A spatio-temporal field  $S \times T$  is a three-dimensional partitioned space, where  $T = \{t_1, t_2, \dots, t_T\}$  is a study period divided into equal-length intervals (e.g., hours, days) and  $\mathcal{S} = \{s_{(0,0)}, \dots, s_{(M,N)}\}$  is a  $M \times N$  two-dimension spatial grid partitioned from the study area. A set of socio-environmental features  $F$  are observed over  $S \times T$ , which include temporal features  $F_t \in \mathbb{R}^T$  (e.g., day of week, holiday), spatial features  $F_s \in \mathbb{R}^{M \times N}$  (e.g., total road length, speed limit), and spatiotemporal features  $F_{st} \in \mathbb{R}^{M \times N \times T}$  (e.g., traffic volume, rainfall amount). A risk score  $y \in \mathbb{R}^{M \times N \times T}$  is a user-defined attribute over  $S \times T$  measuring the risk level of each spatiotemporal location (e.g., number of crimes, injuries of traffic accidents).  $y(s, t) > 0$  when any events occurred in  $(s, t)$ , and equals 0 when no events occurred.
Given the socio-environmental features  $F$  and risk score  $y$  for all the locations in time window  $\{t_1, t_2, \dots, t_n\}$ , the urban event ranking problem is to predict the ranking of the top- $k$  locations  $\{s_1, \dots, s_K\} \in S$  in the next time interval  $t_{n+1}$  with the highest risk scores. The objective is to prioritize the ranking quality on the top- $k$  riskiest locations. As a basic assumption of our problem, there exists spatial and temporal autocorrelation among locations in  $F$ , meaning nearby locations tend to have more correlated values. In addition, events are sparse so  $y$  is 0 for the majority of the locations. A detailed example of data attributes and feature generation steps can be found in the supplementary materials Appendix A.
2.2 Stochastic Optimization of NDCG in Event Ranking Problem
In the field of recommendation systems, learning to rank is substantially studied, and NDCG is widely used as the metric to measure the ranking quality of the foremost importance. In the following part, we use the terminologies from the event forecasting problem to define NDCG in our problem setting. For a ranked list of locations  $s \in S$  in a period  $t \in T$ , the NDCG score is computed as by:
$$
\mathrm {N D C G} _ {t} = \frac {1}{Z _ {t}} \sum_ {s \in S _ {t}} \frac {2 ^ {y _ {s}} - 1}{\log_ {2} (1 + \mathrm {r} (s))}, \tag {1}
$$
where  $y_{s}$  is the risk score of the location  $s$ ,  $\mathrm{r}(s)$  denotes the rank of location  $s$  in the studied spatial domain  $S_{t}$ , and  $Z_{t}$  is the Discounted Cumulative Gain (DCG) score [17] of the perfect ranking of locations for time period  $t$ . However, the rank operator in NDCG is non-differentiable in terms of model parameters, and thus cannot be optimized directly. A popular solution is to approximate the rank operator with smooth functions and then optimize its surrogates [34][28][29] as shown in Eq. 2.
$$
\bar {g} (\mathbf {w}; \mathbf {x}, \mathcal {S} _ {t}) = \sum_ {s ^ {\prime} \in \mathcal {S} _ {t}} \ell \left(h _ {t} \left(s ^ {\prime}; \mathbf {w}\right) - h _ {t} (\mathbf {s}; \mathbf {w})\right), \tag {2}
$$
where rank operator  $r(s)$  in NDCG is approximated by a differentiable surrogate function  $\ell(\cdot)$ , and the squared hinge loss  $\ell(x) = \max(0, x + c)^2$  is commonly used [38]. In this way, the model parameters  $\mathbf{w}$  can be updated by a gradient-based optimizer. We can maximize over  $L(\mathbf{w})$ :
$$
\max  _ {\mathbf {w} \in \mathbb {R} ^ {d}} L (\mathbf {w}) := \frac {1}{| T |} \sum_ {t = 1} ^ {T} \sum_ {s \in \mathcal {S} _ {t} ^ {+}} \frac {2 ^ {y _ {s} ^ {t}} - 1}{Z _ {t} \log_ {2} (\bar {g} (\mathbf {w} ; \mathbf {x} _ {s} ^ {t} , \mathcal {S} _ {t}) + 1)}. \tag {3}
$$
Figure 1: SpatialRank Architecture. The spatiotemporal features are used to generate adjacency matrices and then embedded by graph convolution layers. We use a fully connected layer to make final predictions. The hybrid objective function is combined with NDCG loss and local NDCG loss.
where  $\mathbf{s}^t\in S_t^+$  denotes a set of locations with positive risk scores to be considered in the objective function. In this way, the optimization solution on NDCG can be directly applied to the event ranking problem.
Directly optimizing NDCG as described above might provide a solution to our problem but fails to capture the spatiotemporal autocorrelation in the data. Also, since NDCG is a global measure, the model might not be able to learn how to rank nearby locations with highly correlated features correctly. In this section, we present our SpatialRank method to address these limitations. Figure. 1 demonstrates the proposed model architecture.
3.1 The Deep Learning Model
Many recent deep learning models for spatiotemporal could be used as the backbone architecture of our SpatialRank method. A key idea in these methods is to model the correlations between locations using a graph, with locations as nodes and edge weights representing the strength of correlations, and extract latent spatial dependency information through graph convolution layers [37] [5]. We follow a similar idea to build the overall network architecture, where spatial and spatiotemporal features are fed into graph convolutional layers. Then the extracted latent representations are concatenated with the temporal features and fed into LSTM layers to capture temporal representations before the final output layer.
A key novelty in our SpatialRank network is the design of a time-guided and traffic-aware graph-generating process for the graph convolution layers. Prior works typically use Pearson's correlation coefficients or similar measures of features (e.g., traffic volume, accident counts) between locations as their correlation strengths and pre-compute a time invariant adjacency matrix of locations [41]. In fact, studies [10][24] demonstrate that the influence of traffic conditions on events varies over different periods. Inspired by a related work [39] on a different problem, we use historical events to generate a static graph and learn a time-variant graph from  $F_{ST}$  (e.g., traffic volume) in each time interval. Intuitively, we use  $F_{T}$  (e.g. hour of the day) to learn the weights of dynamic graph vs. static graph to be considered in the graph convolution. The key equations of generating the adaptive graph adjacency are shown below:
$$
Z _ {1} = \tanh  \left(\alpha E _ {1} W _ {1}\right) \tag {4}
$$
$$
Z _ {2} = \tanh  \left(\alpha E _ {2} W _ {2}\right) \tag {5}
$$
$$
\mathcal {A} _ {\text {d y n a m i c}} = \operatorname {R e L U} \left(\tanh  \left(\alpha \left(Z _ {1} Z _ {2} ^ {T} - Z _ {2} Z _ {1} ^ {T}\right)\right)\right) \tag {6}
$$
$$
\beta = \operatorname {s i g m o i d} \left(F _ {T} W _ {3}\right) \tag {7}
$$
$$
\mathcal {A} = \beta \mathcal {A} _ {\text {d y n a m i c}} + (1 - \beta) \mathcal {A} _ {\text {s t a t i c}} \tag {8}
$$
Where  $W_{1}, W_{2}$ , and  $W_{3}$  are learnable parameters.  $E_{1}$  and  $E_{2}$  are randomly initialized node embeddings that can be learned during training. We represent those embeddings by the spatiotemporal features  $F_{ST}$  (e.g., traffic volume) to reveal the underlying dynamic connections between nodes. The subtraction and ReLU activation function in Eq. (6) lead to the asymmetric property of  $A_{dynamic}$ .  $\alpha$  is a hyperparameter to control the saturation rate.  $A_{static}$  is a pre-calculated adjacency matrix before training the model by computing the Pearson correlation coefficient between the risk scores of locations (nodes), where the correlation between node  $i$  and node  $j$ ,  $a_{ij} = \frac{\sum_{t}(y_i - \bar{y}_i)(y_j - \bar{y}_j)}{\sqrt{\sum_{t}(y_i - \bar{y}_i)^2(y_j - \bar{y}_j)^2}}$ . The final adjacency matrix  $A$  is the weighted sum of  $A_{dynamic}$  and  $A_{static}$ , and the weight  $\beta$  is learned by a sigmoid activation function in Eq. 7 from the linear transformation of temporal feature  $F_{T}$ . Intuitively, the static adjacency matrix indicates a baseline correlation between different locations and it is pre-computed before training. This is also what most of the related work has been done. However, inspired by many observations from related studies [10][40], it is evident that we think this correlation is not always constant. Therefore, we use locations' time-variant features to construct a new adjacency matrix, and this dynamic adjacency matrix varies with time. We learn the parameters in this dynamic matrix during the network training process. Finally, we combine the static and the learned dynamic matrices through a learned weight  $\beta$ . In this way, a combined adjacency matrix can be treated as an adaptation from a static adjacency matrix considering the influence of other features during different periods. Finally, extracted embeddings are fed into LSTM layers and make final predictions of the risk scores  $y$  for each location through a fully connected layer.
3.2 Local Ranking Metric and Hybrid Loss function
As previously mentioned, ranking-based metrics such as NDCG are not designed to handle spatial correlations. According to the first law of geography [35], nearby locations may share very similar socio-environmental attributes, thus it is challenging to rank neighboring locations correctly. However, the locations nearby with uncertain event patterns are worthy to be focused on so that more potential events can be discovered. Moreover, using NDCG on event ranking problems can cause overconcentrating on top-ranked locations and sacrificing prediction accuracy on other locations due to lower priority. To address those issues, we design a novel local ranking measurement named Local Normalized Discounted Cumulative Gain (L-NDCG) to measure spatially local ranking quality over every sub-region of the study area. The L-NDCG is calculated as:
$$
\text {L - N D C G} = \frac {1}{T \left| \mathcal {S} _ {t} \right|} \sum_ {t = 1} ^ {T} \sum_ {s ^ {t} \in \mathcal {S} _ {t}} \sum_ {\hat {s} \in \mathcal {N} \left(s ^ {t}\right)} \frac {2 ^ {y _ {\hat {s}} ^ {t}} - 1}{Z _ {\mathcal {N} \left(s ^ {t}\right)} ^ {t} \log_ {2} \left(r (\hat {s} , \mathcal {N} \left(s ^ {t}\right)) + 1\right)}. \tag {9}
$$
We use the same terminologies from Eq. 3. The unique part is that we compute an NDCG score for every location in the study area based on the local ranking of a subset of locations  $\mathcal{N}(s^t)$ , where  $\mathcal{N}$  is a neighborhood of location  $s^t$ . We define the  $\mathcal{N}$  as the Euclidean distance of coordinates smaller than  $R$  in this work.  $\mathcal{N}$  can be defined in other ways depending on the need of the problem. Essentially, L-NDCG is the average of NDCG scores for all subsets of locations. In this way, a few stationed hot-spot locations only take considerably large weight in their own NDCG scores and cannot over-influence the overall L-NDCG score. L-NDCG emphasizes ranking correctly on each subset of locations, and a greater L-NDCG score indicates that relatively more important locations can be distinguished from their nearby locations in a small region. Similar to optimizing NDCG, the ranking operator of L-NDCG is non-differentiable, thus we optimize its surrogates instead.
$$
\max  _ {\mathbf {w} \in \mathbb {R} ^ {d}} L (\mathbf {w}) := \frac {1}{T \left| \mathcal {S} _ {t} ^ {+} \right|} \sum_ {t = 1} ^ {T} \sum_ {s ^ {t} \in \mathcal {S} _ {t} ^ {+}} \sum_ {\hat {s} \in \mathcal {N} (s ^ {t})} \frac {2 y _ {\hat {s}} ^ {t} - 1}{Z _ {\mathcal {N} (s ^ {t})} ^ {t} \log_ {2} (\bar {g} (\mathbf {w} ; \mathbf {x} _ {\hat {s}} ^ {t} , \mathcal {N} (s ^ {t})) + 1)}. \tag {10}
$$
Where  $g(\mathbf{w};\mathbf{x}_{\hat{s}}^{t},\mathcal{N}(s^{t}))$  is a surrogate loss function similar to Eq. 2, and  $\mathcal{N}(s^t)$  is a set of locations. Finally, to learn a trade-off between locally ranking quality and globally ranking quality we design a hybrid objective function consisting of both NDCG and L-NDCG
$$
L o s s = (1 - \sigma) \mathrm {N D C G} + \sigma \mathrm {L} - \mathrm {N D C G} \tag {11}
$$
Where  $\sigma$  is a hyperparameter controlling the preference between NDCG and L-NDCG.
3.3 Importance-based Location Sampling with Spatial Filtering
To bridge the gap between capturing spatial correlations in a list of locations and optimizing the quality of predicted ranking, we propose a novel importance-based location sampling strategy with spatial filtering. Specifically, we first design an importance measure function to assign higher weights to locations with larger errors in predictions and higher ranking priority. Secondly, we sample locations based on the weights assigned by importance-measuring, so that important locations have a higher probability to be sampled. Afterward, only losses from sampled locations will be calculated in the objective function and considered during the optimization. In this way, the model pays attention to more important locations. Thirdly, we adjusted the importance scores every epoch so that the model learns to focus on different locations adaptively. Lastly, spatial filters are applied to smooth the importance scores in each epoch to achieve spatial-aware sampling. This allows nearby locations to be sampled in the same batch with high probability to help the model learn how to rank them. The training process is shown in algorithm 1
Algorithm 1: SpatialRank Training
Input: feature tensor  $F$  , event risk scores  $y$  , hyperparameter  $\sigma$  , standard deviation  $\lambda$  Output: learned model  $f_{\theta}$    
1 Initialize probability set  $P = \{p_1,p_2,\dots,p_l\} \in L;$    
for each epoch do   
3 Compute  $\hat{y} = f_{\theta}(x)$    
4 Compute lossNDCG  $\equiv$  WeightedLoss(y,  $\hat{y}$  P)   
5 Compute losslocal  $\equiv$  LocalWeightedLoss(y,  $\hat{y}$  P)   
6 losshybrid  $= (1 - \sigma)\cdot$  lossNDCG  $^+$ $\sigma$  .losslocal   
7 for  $s\in S$  do   
8  $E_{s} = \frac{1}{T}\sum_{t}\frac{2|y_{s}^{t} - \hat{y}_{s}^{t}| - 1}{\log_{2}(1 + r(y_{s}^{t}))}$    
9  $E_{s'}^* = \sum_s\frac{E_s}{2\pi\lambda^2} e^{-\frac{x^2}{2\lambda^2}}$  for  $s\in S$ $x = dist(s',s)$    
10 Update  $\mathrm{P} =$  normorlize(E')   
11 Compute gradient  $\nabla f(\theta)$  by losshybrid   
12 Update  $\theta$  by  $\nabla f(\theta)$
Algorithm 1 shows the details of the importance-based sampling mechanism. The inputs include feature tensor  $F$ , event risk scores  $y$ , hyperparameter  $\sigma$ , and Gaussian standard deviation  $\lambda$ . The output is a learned model. The algorithm starts with initializing a set of equal-importance scores, which means the probabilities of locations being sampled are equal in the first epoch. Line 3 is forward propagation with current model parameters. Line 4 calculates the weighted NDCG losses given true label  $y$ , predicted labels  $\hat{y}$ , and current importance scores for locations.  $Loss_{NDCG}$  is computed by Eq. 9, where importance scores  $\{p_1, p_2, \dots, p_s\} \in S$  are normalized and treated as weights. Line 6 is our novel hybrid loss function discussed in Eq 10 to leverage the local ranking quality. The key step is to update the importance scores set based on current prediction errors and true ranking in Lines 7-8. We design a score function shown in Line 8 to leverage the errors made in predictions and the priority of true ranking for each location.
$$
E _ {s} = \frac {1}{T} \sum_ {t} \frac {2 \left| y _ {s} ^ {t} - \hat {y} _ {s} ^ {t} \right| - 1}{\log_ {2} \left(1 + r \left(y _ {s} ^ {t}\right)\right)} \tag {12}
$$
where  $r(\cdot)$  denotes a ranking function of the  $i$ -th location in the study area, and  $|y_s^t -\hat{y}_s^t|$  is the absolute difference between predicted injuries and true injuries. Basically, a higher score  $E_{l}$  indicates that