SlowGuess/NeurIPS_2023_OCR huggingface dataset:
there are larger prediction errors and higher true ranking in this location  $s$ . Note that smaller  $r(y_s^t)$  denotes a higher true ranking. To capture their geographical connections, we map the list of locations to their original locations on the grids, and then we apply a Gaussian filter to smooth the score distribution spatially in Line 9.  $\lambda$  is the standard deviation. In this way, the model can learn better spatially correlated patterns, and avoid over-focusing on a few standalone locations but ignoring the area nearby. Next, the smoothed scores  $E^*$  are normalized into  $P$ , where  $\sum_{s} P_s = 1$ . Finally, we compute the gradient  $\nabla f(\theta)$  by  $loss_{hybrid}$  and update model parameters  $\theta$ . After a few iterations, we obtain the learned model  $f_\theta$ .
3.4 Complexity Analysis
In each iteration complexity of SpatialRank, we need to conduct forward propagation  $h_t(s; \mathbf{w}), \forall s \in S_t^+ \cup S_t$  and back-propagation for computing  $\nabla h_t(s; \mathbf{w}), \forall s \in S_t^+ \cup S_t$ . The complexity of forwarding is  $S = \sum_{t \in T} (|\mathcal{S}_t^+| |\mathcal{N}(s) + |\mathcal{S}_t|) d \leq O(TSd)$ , where  $d$  is the model size. To compute local_NDCG part, an extra cost on neighbor querying is needed to replace location list size on its complexity. The cost of computing  $\bar{g}(\mathbf{w}; \mathbf{x}, S_t)$  is  $\sum_{t \in T} |S_t^+| |\mathcal{S}_t| \leq O(TS^2)$  for NDCG and  $\sum_{t \in T} |\mathcal{S}_t^+| \mathcal{N}^2(s) \leq O(TS\mathcal{N}^2(s))$  for local NDCG. The size of  $\mathcal{N}^2(s)$  is small, therefore the total cost is reduced to  $O(TSd + TS^2)$ .
We perform comprehensive experiments on three real-world traffic accident and crime datasets from Chicago $^2$  and the State of Iowa $^3$ . Experiment results show that our proposed approach substantially outperforms state-of-art baselines on three datasets by up to  $12.7\%$ ,  $7.8\%$ , and  $4.2\%$  in NDCG respectively. The case study and results on the state of Iowa are shown in Appendix C.
Data. In the Chicago dataset, we collect data from the year 2019 to the year 2021. The first 18 months of this period are used as the training set, and the last 6 months of 2020 are used as the validating set. The year 2021 is used as a testing set. The area of Chicago is partitioned by  $500\mathrm{m} \times 500\mathrm{m}$  square cells and converted to a grid with the size of  $64 \times 80$ . For the crime dataset, we use total crimes as the risk score. For accident datasets, we use the number of injuries as the risk score. w Baselines. First, we use daily Historical Average (HA). Next, we consider popular machine-learning methods including Long Short-term Memory (LSTM) [16], and Convolutional LSTM Network (ConvLSTM) [31]. Thirdly, we compare with recent methods such as GSNet [37], Hetero-ConvLSTM[40], and HintNet [5]. Moreover, we compared our optimization approach with other NDCG optimization solutions including Cross Entropy (CE), ApproxNDCG [28], and SONG [29]. The details of the baselines are described in Appendix C.
Metrics. To measure the ranking quality of foremost locations, we test  $K \in [30, 40, 50]$  on the test data. We use the metrics including NDCG, L-NDCG, and top-K precision (Prec)[23]. We report the average performance and standard deviation over 3 runs for three datasets.
4.1 Performance Comparison
In table 1, we can observe that our SpatialRank significantly outperforms other compared baselines in both datasets. We observe a similar trend among all metrics. On both datasets, Hetero-ConvLSTM and HintNet achieve similar results and outperform general machine learning methods such as LSTM and ConvLSTM. GSNet is designed on a dataset with a smaller grid size, thus performing worse on our larger grid. To study the effectiveness of learning the appropriate graph, we set the learning parameter  $\beta$  as a fixed ratio of 0.5 in SpatialRank#. Oppositely,  $\beta$  is a parameter to be learned based on temporal features in SpatialRank. In the Table 1 and Table. 2, each * indicates that the performance improvement of the proposed method over this baseline is statistically significant based on the student t-test with  $\alpha = 0.05$  over three runs. The results show that the design of the time-aware graph convolution is able to improve performance and capture more dynamic variations in the graph, therefore performing better over different top-k rankings.
2 https://data.cityofchicago.org/Transportation/Traffic-Crashes-Crashes/85ca-t3if
3 https://icat.iowadot.gov/
TABLE 1: PERFORMANCE COMPARISON
CHICAGO ACCIDENT	K=30	K=40	K=50
NDCG	PREC	L-NDCG	NDCG	PREC	L-NDCG	NDCG	PREC	L-NDCG
HA	.214±0	.332±0	.502±0	.225±0	.322±0	.497±0	.235±0	.316±0	.493±0
LSTM	.215±2%.*	.392±2%.*	.519±3%.*	.225±1%.*	.380±2%.*	.543±3%.*	.249±1%.*	.368±2%.*	.544±3%*
CONVLSTM	.225±5%.*	.410±4%.*	.558±3%.*	.236±1%.*	.388±4%.*	.563±8%.*	.252±1%.*	.366±2%.*	.540±8%*
GSNET	.194±1%.*	.371±2%.*	.493±5%.*	.201±2%.*	.371±2%.*	.517±5%.*	.231±1%.*	.337±3%.*	.499±3%*
HETERO-CONVLSTM	.229±2%.*	.401±1%.*	.557±2%.*	.240±1%.*	.395±4%.*	.564±3%.*	.255±3%.*	.375±2%.*	.551±3%*
HINTNET	.228±1%.*	.400±2%.*	.555±3%.*	.238±2%.*	.390±3%.*	.569±3%.*	.256±1%.*	.373±4%.*	.561±8%*
SPATIALRANK#	.250±2%.*	.438±3%.*	.591±3%.*	.256±1%.*	.409±2%.*	.593±2%.*	.271±2%.*	.394±2%.*	.585±1%
SPATIALRANK	.257±1%.*	.444±3%.*	.621±4%.*	.268±1%.*	.420±1%.*	.614±2%.*	.278±3%.*	.403±1%.*	.599±1%
CHICAGO CRIME	K=30	K=40	K=50
NDCG	PREC	L-NDCG	NDCG	PREC	L-NDCG	NDCG	PREC	L-NDCG
HA	.237±0	.348±0	.514±0	.250±0	.333±0	.506±0	.259±0	.322±0	.449±0
LSTM	.246±1%.*	.327±2%.*	.517±3%.*	.257±1%.*	.329±2%.*	.521±3%.*	.262±3%.*	.314±5%.*	.512±3%*
CONVLSTM	.313±2%.*	.415±4%.*	.617±4%.*	.325±2%.*	.404±1%.*	.607±6%.*	.333±2%.*	.387±4%.*	.599±3%*
GSNET	.283±3%.*	.388±2%.*	.584±3%.*	.296±2%.*	.374±2%.*	.565±3%.*	.299±2%.*	.335±4%.*	.568±3%*
HETERO-CONVLSTM	.346±3%.*	.468±3%.*	.657±4%.*	.365±1%.*	.452±3%.*	.642±6%.*	.374±4%.*	.433±4%.*	.638±4%*
HINTNET	.342±3%.*	.468±3%.*	.661±4%.*	.358±3%.*	.448±4%.*	.631±6%.*	.369±2%.*	.434±2%.*	.628±4%*
SPATIALRANK#	.361±2%.*	.484±1%.*	.670±4%.*	.376±4%.*	.463±3%.*	.655±7%.*	.387±1%.*	.446±1%.*	.651±7%
SPATIALRANK	.373±2%.*	.491±3%.*	.665±4%.*	.380±2%.*	.467±5%.*	.647±6%.*	.392±2%.*	.446±3%.*	.644±6%
$\beta = 0.5$ $\% o: \times 10^{-3}$
TABLE 2: OPTIMIZATION COMPARISON
CHICAGO ACCIDENT	K=30	K=40	K=50
NDCG	PREC	L-NDCG	NDCG	PREC	L-NDCG	NDCG	PREC	L-NDCG
CE	.232±1‰	.424±1‰	.588±2‰	.253±1‰	.415±2‰	.588±2‰	.266±1‰	.400±1‰	.580±2‰
APPROXDCG	.240±2‰	.426±2‰	.597±3‰	.255±1‰	.415±1‰	.591±6‰	.264±1‰	.398±2‰	.575±3‰
SONG	.240±1‰	.438±2‰	.610±6‰	.254±2‰	.417±1‰	.600±1‰	.267±2‰	.400±4‰	.575±1‰
SPATIALRANK	.257±1‰	.444±3‰	.621±4‰	.268±1‰	.420±1‰	.614±2‰	.278±3‰	.403±1‰	.599±1‰
CHICAGO CRIME	K=30	K=40	K=50
NDCG	PREC	L-NDCG	NDCG	PREC	L-NDCG	NDCG	PREC	L-NDCG
CE	.362±1‰	.475±1‰	.653±3‰	.378±1‰	.455±1‰	.646±1‰	.386±1‰	.441±2‰	.644±2‰
APPROXDCG	.344±4‰	.455±3‰	.637±10‰	.353±1‰	.435±2‰	.658±5‰	.365±3‰	.421±4‰	.619±6‰
SONG	.364±2‰	.484±3‰	.660±5‰	.379±4‰	.466±3‰	.456±1‰	.390±2‰	.450±3‰	.649±6‰
SPATIALRANK	.373±2‰	.491±3‰	.665±4‰	.380±2‰	.467±5‰	.647±6‰	.392±2‰	.446±3‰	.644±6‰
$\% \text{：}\times 10^{-3}$
4.2 Optimization Comparison
To evaluate the effectiveness of our proposed hybrid objective function and importance-based location sampling, we perform experiments on the same network architecture but with different optimization solutions including Cross Entropy (CE), ApproxNDCG, and SONG. The results are shown in table 2. Methods designed to optimize NDCG consistently perform better than Cross Entropy. SpatialRank substantially outperforms SONG and ApproxNDCG and made a noticeable improvement on L-NDCG as it is considered in the objective function.
We examine the effects of tuning hyper-parameter  $\sigma$  in the hybrid loss function. We present the results in table 3. Recall that  $\sigma$  decides the ratio of L-NDCG takes in the objective function.  $\sigma = 0$  means L-NDCG is not considered in the objective function. Starting from  $\sigma = 0$ , we observe a trend that the overall performance improves steadily while  $\sigma$  goes up, and it reaches the best performance
TABLE 3: ABLATION STUDY
CHICAGO ACCIDENT	K=30	K=40	K=50
NDCG	PREC	L-NDCG	NDCG	PREC	L-NDCG	NDCG	PREC	L-NDCG
σ = 0.0	0.251	0.439	0.610	0.263	0.407	0.612	0.274	0.289	0.605
σ = 0.05	0.239	0.416	0.559	0.254	0.394	0.600	0.266	0.386	0.597
σ = 0.1	0.256	0.443	0.621	0.268	0.421	0.608	0.276	0.400	0.599
σ = 0.2	0.250	0.444	0.616	0.261	0.411	0.605	0.271	0.394	0.603
σ = 0.3	0.234	0.422	0.606	0.243	0.388	0.59	0.253	0.374	0.591
CHICAGO CRIME	K=30	K=40	K=50
NDCG	PREC	L-NDCG	NDCG	PREC	L-NDCG	NDCG	PREC	L-NDCG
σ = 0.0	0.366	0.489	0.661	0.378	0.464	0.644	0.385	0.445	0.642
σ = 0.05	0.366	0.489	0.659	0.378	0.467	0.649	0.382	0.447	0.643
σ = 0.1	0.371	0.494	0.665	0.383	0.467	0.644	0.391	0.450	0.642
σ = 0.2	0.356	0.474	0.654	0.367	0.448	0.643	0.372	0.429	0.623
σ = 0.3	0.345	0.457	0.658	0.361	0.445	0.641	0.373	0.431	0.631
Figure 2: Comparison of cross-K function in Chicago Accident.
at  $\sigma$  equals 0.1 in the accident and crime dataset. It indicates that considering a reasonable weight of L-NDCG in the objective function boosts overall performance. These results demonstrate the importance of considering local ranking because NDCG, L-NDCG, and precision can be all improved.
4.4 Cross-K function
We use the Cross-K function[14] with Monte Carlo Simulation[33] to evaluate the accuracy of predicted locations. The Cross-K function measures the spatial correlation between the predicted locations and true locations. Specifically, we calculate the average density of predictions within every distance  $d$  of a true event in each day as shown in Eq. 13:
$$
\hat {K} (d) = \lambda_ {j} ^ {- 1} \sum_ {i \neq j} I \left(d _ {i j} \leq d\right) / n, \tag {13}
$$
where  $\lambda$  is global density of event  $j$ , and  $I()$  is an identity function which equals one if real distance  $d_{ij}$  is smaller than  $d$ , else equals zero.  $n$  is the number of events  $i$ . The results are shown in Figure 2. The grey curve represents the complete spatial randomness and we use it as a reference baseline. Higher is better. Our SpatialRank achieves the best predictions in both datasets, which indicates that the predictions of SpatialRank are significantly spatially correlated with ground truth. The similar results on the other two datasets are shown in Appendix C in the supplementary materials.
Additional Results and a Case Study to demonstrate successful prediction examples are included in the Supplementary materials.
Urban Event forecasting has been widely studied in the past few decades. Most early studies [13] [21] [6][19] rely on small-scale and single-source datasets with limited types of features, thus the prediction accuracy is limited. Notably, Zhou et al. [42] proposed a differential time-varying graph convolution network capturing traffic changes and improving prediction accuracy. Similarly, Wang et al. [37] proposed GSNet with a geographical module and a weighted loss function to capture semantic spatial-temporal correlations among regions and solve data sparsity issues. Addressing the issue of spatial heterogeneity, Yuan et al. [40] proposed Hetero-ConvLSTM to leverage an ensemble of predictions from models learned from pre-selected sub-regions. Furthermore, An et al. [5] proposed HintNet partitions the study area hierarchically and transfers learned knowledge over different regions to improve performance. However, most existing models rely on optimizing cross-entropy and the objective is to make accurate predictions on every location. Learning to rank is an extensively studied area in recommendation systems and search engines [22]. NDCG is a widely adopted metric to measure ranking quality. The prominent class of methods involves approximating ranks in NDCG using smooth functions and subsequently optimizing the resultant surrogates. Taylor et al. [34] tries to use rank distributions to smooth NDCG, but suffers from high computational cost. Similarly, Qin et al. [28] approximates the indicator function by a generalized sigmoid function with a top-k variant. Noticeably, Qiu et al. [29] develop stochastic algorithms optimizing the surrogates for NDCG and its top-K variant. Although it is possible to directly apply the existing ranking methods to the urban event ranking problem, the performance tends to be unsatisfactory as these methods commonly assume independence between items and queries and lack the ability to handle spatiotemporal autocorrelation in the data.
In this work, we formulate event forecasting as a location ranking problem. We propose a novel method SpatialRank to learn from spatiotemporal data by optimizing NDCG surrogates. To capture dynamic spatial correlations, we design an adaptive graph convolution layer to learn the graph from features. Furthermore, we propose a hybrid loss function to capture potential risks around hot-spot regions, and a novel ranking-based importance sampling mechanism to leverage the importance of each location considered during the model training. Extensive experimental results on three real-world datasets demonstrate the superiority of SpatialRank compared to baseline methods.
Limitations: the model's performance might be affected by other properties of data, such as spatial heterogeneity and sparsity. We observe less improvement over baselines on the Iowa dataset, partially due to that the data is sparser over a large area with heterogeneity. These are issues addressed by some of the prior work and can be addressed in our future work. In addition, the new algorithm increases the training time complexity due to the sampling steps. This is acceptable due to the relatively small number of locations and time periods in urban event datasets but may require extra work to generalize to large datasets.
T. Yang was partially supported by NSF Career Award 18844403, NSF Fair AI Grant 2147253, NSF RI Grant 2110545.
[1] Chicago Police Department, https://home.chicagopolice.org/cpd-expands-smart-policing-technology-to-support-strategic-deployment-and-cta-safety/. 2020.
[2] Fox News, https://www.foxnews.com/us/us-murder-rate-continued-grim-climb-i2021-new-fbi-estimates-show. 2022.
[3] Police Executive Research Forum, https://www.policeforum.org/workforcemarch2022. 2022.
[4] J. Abellán, G. López, and J. De Oña. Analysis of traffic accident severity using decision rules via decision trees. Expert Systems with Applications, 40(15):6047-6054, 2013.
[28] T. Qin, T.-Y. Liu, and H. Li. A general approximation framework for direct optimization of information retrieval measures. Information Retrieval, 13(4):375-397, 2010.
[29] Z. Qiu, Q. Hu, Y. Zhong, L. Zhang, and T. Yang. Large-scale stochastic optimization of ndcg surrogates for deep learning with provable convergence. In International Conference on Machine Learning, 2022.
[30] H. Ren, Y. Song, J. Wang, Y. Hu, and J. Lei. A deep learning approach to the citywide traffic accident risk prediction. 2018 21st International Conference on Intelligent Transportation Systems (ITSC), pages 3346-3351, 2018.
[31] X. Shi, Z. Chen, H. Wang, D.-Y. Yeung, W.-k. Wong, and W.-c. Woo. Convolutional LSTM network: A machine learning approach for precipitation nowcasting. 2015.
[32] R. Swezey, A. Grover, B. Charron, and S. Ermon. Pirank: Scalable learning to rank via differentiable sorting. Advances in Neural Information Processing Systems, 34, 2021.
[33] R. Tao and J.-C. Thill. Flow cross k-function: a bivariate flow analytical method. International journal of geographical information science : IJGIS, 33(10):2055-2071, 2019.
[34] M. Taylor, J. Guiver, S. Robertson, and T. Minka. Softrank: optimizing non-smooth rank metrics. In Proceedings of the 2008 International Conference on Web Search and Web Data Mining, pages 77-86, 2008.
[35] W. R. Tobler. A computer movie simulating urban growth in the Detroit region. *Economic geography*, 46:234-240, 1970.
[36] A. Vahedian, X. Zhou, L. Tong, W. N. Street, and Y. Li. Predicting urban dispersal events: A two-stage framework through deep survival analysis on mobility data. 2019.
[37] B. Wang, Y. Lin, S. Guo, and H. Wan. Gsnet: Learning spatial-temporal correlations from geographical and semantic aspects for traffic accident risk forecasting. In 2021 AAAI Conference on Artificial Intelligence (AAAI'21), 2021.
[38] M. Wu, Y. Chang, Z. Zheng, and H. Zha. Smoothing deg for learning to rank: a novel approach using smoothed hinge functions. In Proceedings of the 18th ACM conference on information and knowledge management, CIKM '09, pages 1923-1926. ACM, 2009.
[39] Z. Wu, S. Pan, G. Long, J. Jiang, X. Chang, and C. Zhang. Connecting the dots: Multivariate time series forecasting with graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 753-763, 2020.
[40] Z. Yuan, X. Zhou, and T. Yang. Hetero-convlstm: A deep learning approach to traffic accident prediction on heterogeneous spatio-temporal data. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 984–992, 2018.
[41] Y. Zhang, Y. Li, X. Zhou, X. Kong, and J. Luo. Trafficgan: Off-deployment traffic estimation with traffic generative adversarial networks. 2019 IEEE International Conference on Data Mining (ICDM), pages 1474-1479, 2019.
[42] Z. Zhou, Y. Wang, X. Xie, L. Chen, and C. Zhu. Foresee urban sparse traffic accidents: A spatiotemporal multi-granularity perspective. IEEE Transactions on Knowledge and Data Engineering, pages 1-1, 2020.