Title: SpatialRank: Urban Event Ranking with NDCG Optimization on Spatiotemporal Data
Venue: NeurIPS 2023
Accepted: yes
Keywords: urban event

NDCG optimization

ranking

traffic accident

crime

spatiotemporal data
Abstract: The problem of urban event ranking aims at predicting the top-$k$ most risky locations of future events such as traffic accidents and crimes. This problem is of fundamental importance to public safety and urban administration especially when limited resources are available. The problem is, however, challenging due to complex and dynamic spatio-temporal correlations between locations, uneven distribution of urban events in space, and the difficulty to correctly rank nearby locations with similar features. Prior works on event forecasting mostly aim at accurately predicting the actual risk score or counts of events for all the locations. Rankings obtained as such usually have low quality due to prediction errors. Learning-to-rank methods directly optimize measures such as Normalized Discounted Cumulative Gain (NDCG), but cannot handle the spatiotemporal autocorrelation existing among locations. Due to the common assumption that items are independent. In this paper, we bridge the gap by proposing a novel spatial event ranking approach named SpatialRank. SpatialRank features  adaptive graph convolution layers that dynamically learn the spatiotemporal dependencies across locations from data. In addition, the model optimizes through surrogates a hybrid NDCG loss with a spatial component to better rank neighboring spatial locations. We design an importance-sampling with a spatial filtering algorithm to effectively evaluate the loss during training. Comprehensive experiments on three real-world datasets demonstrate that SpatialRank can effectively identify the top riskiest locations of crimes and traffic accidents and outperform state-of-art methods in terms of NDCG by up to 12.7%.

SpatialRank: Urban Event Ranking with NDCG Optimization on Spatiotemporal Data

Abstract

The problem of urban event ranking aims at predicting the topk most risky locations of future events such as traffic accidents and crimes. This problem is of fundamental importance to public safety and urban administration especially when limited resources are available. The problem is, however, challenging due to complex and dynamic spatio-temporal correlations between locations, uneven distribution of urban events in space, and the difficulty to correctly rank nearby locations with similar features. Prior works on event forecasting mostly aim at accurately predicting the actual risk score or counts of events for all the locations. Rankings obtained as such usually have low quality due to prediction errors. Learning-to-rank methods directly optimize measures such as Normalized Discounted Cumulative Gain (NDCG), but cannot handle the spatiotemporal autocorrelation existing among locations. In this paper, we bridge the gap by proposing a novel spatial event ranking approach named SpatialRank. SpatialRank features adaptive graph convolution layers that dynamically learn the spatiotemporal dependencies across locations from data. In addition, the model optimizes through surrogates a hybrid NDCG loss with a spatial component to better rank neighboring spatial locations. We design an importance-sampling with a spatial filtering algorithm to effectively evaluate the loss during training. Comprehensive experiments on three real-world datasets demonstrate that SpatialRank can effectively identify the top riskiest locations of crimes and traffic accidents and outperform state-of-the-art methods in terms of NDCG by up to 12 . 7% .

1 Introduction

Given a risk score defined based on historical urban events (e.g., crimes, traffic accidents) in a study area as well as the socio-environmental attributes (e.g., travel demands, weather, road network) associated with the events, the goal of the urban event ranking problem is to learn a model that can predict the ranking of the topk riskiest locations of future events.

The urban event ranking problem is a widely observed spatiotemporal prediction problem, which has critical applications in public safety, traffic management, and urban planning. For example, the National Highway Traffic Safety Administration [27], estimated a total of 42,939 deaths in motor vehicle traffic crashes in the United States in 2021. Crime control and property loss cost over $2.5 trillion in the United States in 2017 [25]. Chicago Police Department has utilized criminal intelligence analysis and data science techniques to help command staff determine where best to deploy resources [1]. However, according to the Police Executive Research Forum, there were 42 . 7% more resignations among law enforcement but a 3 . 9% decrease in hiring new officers in 2021 compared to 2019 [3]. Meanwhile, the Federal Bureau of Investigation confirms that violent crime in 2020 has surged nearly 30% over 2019 [2]. Therefore, given such growth of crimes and accidents, predicting the riskiest locations of traffic accidents or crimes helps law enforcement stakeholders allocate their limited resources strategically to reduce injuries, deaths, and property losses.

Despite its importance, the urban event ranking problem is technically challenging. First , there exist complex and dynamic spatiotemporal correlations between locations in terms of events and attributes such as traffic conditions and weather changes. Capturing such dynamic relationships is non-trivial. Second , due to the presence of spatial autocorrelation, nearby locations may share very similar socio-environmental attributes. It is thus very challenging to learn a good model that can correctly rank neighboring locations. Finally , urban events are usually sparse in space. Making a prediction with a low average error in count does not ensure a good ranking of the topk locations.

Prior works on urban event prediction employed either classic machine learning methods [15] [4] [7] [9] [11] [12] [18] or deep learning techniques including recurrent neural networks [30] and convolutional neural networks [26][36]. Recently, Li et al. [20] proposed a cross-region hypergraph structure network to address the data sparsity issues. Besides, Yuan et al. [40] [5] proposed spatial ensembles to address spatial heterogeneity. The objective of the above methods is to make accurate predictions of the risk or count for all locations. However, urban events are often very sparse in space, which might guide the model to avoid predicting accidents in most locations to obtain a low average error. Such prediction does not truly benefit the users such as police officers. The topk locations derived from such predictions are naturally inaccurate.

Our problem is also relevant to the learning-to-rank problem frequently studied in the field of recommendation systems. State-of-the-art methods in this area typically predict the topk items most likely to be chosen by users through optimizing ranking-based metrics such as the Normalized Discounted Cumulative Gain (NDCG) [8]. The rank operator of NDCG is non-differentiable, and previous studies have made noticeable progress in approximating NDCG by surrogate functions [32] [28] [34] [29]. In our problem, a straightforward adaptation would be to consider the locations as 'items' and each time slot as a 'user'. However, NDCG is not a perfect objective function for our problems as it neither measures the local ranking quality nor considers spatial autocorrelation among locations. Instead, the existing approaches assume items are independent. Therefore, directly applying existing NDCG optimization solutions might not be the best solution to our problem.

In this paper, we bridge the gaps in both fields by formulating the urban event ranking problem as a spatial learning-to-rank problem and solving it by directly optimizing a 'spatial' version of the NDCG measure. We propose SpatialRank , our deep learning model with three novel designs. To efficiently capture the spatial and temporal dependencies, we design an adaptive graph convolution layer that learns the correlations between locations dynamically from features and historical events patterns; to ensure the model balances both the global ranking quality on all the locations and the local ranking quality on subsets of locations, we propose a hybrid loss function combining both NDCG loss and a novel local NDCG loss; to improve the effectiveness of optimizing NDCG surrogates in the spatiotemporal setting of our problem, we design an importance-based location sampling with spatial filtering algorithm to iteratively adjust the weights of each location considered in the objective function, thereby guiding the model to concentrate on learning for more important locations. We conduct comprehensive experiments on three real-world datasets collected from Chicago and the state of Iowa. The results demonstrate that SpatialRank can substantially outperform baselines and achieve better ranking quality.

Our contributions are summarized below:

To the best of our knowledge, this is the first paper to formulate urban event forecasting as a location ranking problem and learn a ranking model by optimizing its ranking quality.

We propose SpatialRank with adaptive graph convolution layers learning from historical event patterns and spatiotemporal features to capture dynamic correlations over time and space.

We propose a hybrid objective function to leverage the trade-off between global ranking quality and local ranking quality.

We propose a ranking-based importance sampling algorithm to adaptively adjust the weights of different locations considered in the objective function of the prediction results from the last training epoch to help the model focus on important locations.

2 Preliminaries

2.1 Formulation of the event ranking problem

A spatio-temporal field S × T is a three-dimensional partitioned space, where T = { t 1 , t 2 , ..., t T } is a study period divided into equal-length intervals (e.g., hours, days) and S = { s (0 , 0) , ..., s ( M,N ) } is a M × N two-dimension spatial grid partitioned from the study area. A set of socio-environmental features F are observed over S × T , which include temporal features F t ∈ R T (e.g., day of week, holiday), spatial features F s ∈ R M × N (e.g., total road length, speed limit), and spatiotemporal features F st ∈ R M × N × T (e.g., traffic volume, rainfall amount). A risk score y ∈ R M × N × T is a user-defined attribute over S × T measuring the risk level of each spatiotemporal location (e.g., number of crimes, injuries of traffic accidents). y ( s, t ) > 0 when any events occurred in ( s, t ) , and equals 0 when no events occurred.

Given the socio-environmental features F and risk score y for all the locations in time window { t 1 , t 2 , .., t n } , the urban event ranking problem is to predict the ranking of the topk locations { s 1 , ..., s K } ∈ S in the next time interval t n +1 with the highest risk scores. The objective is to prioritize the ranking quality on the topk riskiest locations. As a basic assumption of our problem, there exists spatial and temporal autocorrelation among locations in F , meaning nearby locations tend to have more correlated values. In addition, events are sparse so y is 0 for the majority of the locations. A detailed example of data attributes and feature generation steps can be found in the supplementary materials Appendix A.

2.2 Stochastic Optimization of NDCG in Event Ranking Problem

In the field of recommendation systems, learning to rank is substantially studied, and NDCG is widely used as the metric to measure the ranking quality of the foremost importance. In the following part, we use the terminologies from the event forecasting problem to define NDCG in our problem setting. For a ranked list of locations s ∈ S in a period t ∈ T , the NDCG score is computed as by:

where y s is the risk score of the location s , r ( s ) denotes the rank of location s in the studied spatial domain S t , and Z t is the Discounted Cumulative Gain (DCG) score [17] of the perfect ranking of locations for time period t . However, the rank operator in NDCG is non-differentiable in terms of model parameters, and thus cannot be optimized directly. A popular solution is to approximate the rank operator with smooth functions and then optimize its surrogates [34][28][29] as shown in Eq. 2.

where rank operator r ( s ) in NDCG is approximated by a differentiable surrogate function ℓ () , and the squared hinge loss ℓ ( x ) = max(0 , x + c ) 2 is commonly used [38]. In this way, the model parameters w can be updated by a gradient-based optimizer. We can maximize over L ( w ) :

time

F

F

Historical

FST

Historical

FST

Historical

FST

Accident

Accident

Accident

★

Fs

yt+1

GC

GC

GC

GC

FC

FC

FC

NDCG

FC

+

Hybrid Loss

Long Short-Term Memory

yt+1

Local NDCG

FeatureTensor

Graph Convolution

Fully Connected Layer

where s t ∈ S + t denotes a set of locations with positive risk scores to be considered in the objective function. In this way, the optimization solution on NDCG can be directly applied to the event ranking problem.

3 Methodology

Directly optimizing NDCG as described above might provide a solution to our problem but fails to capture the spatiotemporal autocorrelation in the data. Also, since NDCG is a global measure, the model might not be able to learn how to rank nearby locations with highly correlated features correctly. In this section, we present our SpatialRank method to address these limitations. Figure. 1 demonstrates the proposed model architecture.

3.1 The Deep Learning Model

Many recent deep learning models for spatiotemporal could be used as the backbone architecture of our SpatialRank method. A key idea in these methods is to model the correlations between locations using a graph, with locations as nodes and edge weights representing the strength of correlations, and extract latent spatial dependency information through graph convolution layers [37] [5]. We follow a similar idea to build the overall network architecture, where spatial and spatiotemporal features are fed into graph convolutional layers. Then the extracted latent representations are concatenated with the temporal features and fed into LSTM layers to capture temporal representations before the final output layer.

A key novelty in our SpatialRank network is the design of a time-guided and traffic-aware graphgenerating process for the graph convolution layers. Prior works typically use Pearson's correlation coefficients or similar measures of features (e.g., traffic volume, accident counts) between locations as their correlation strengths and pre-compute a time invariant adjacency matrix of locations [41]. In fact, studies [10][24] demonstrate that the influence of traffic conditions on events varies over different periods. Inspired by a related work [39] on a different problem, we use historical events to generate a static graph and learn a time-variant graph from F ST (e.g., traffic volume) in each time interval. Intuitively, we use F T (e.g. hour of the day) to learn the weights of dynamic graph vs. static graph to be considered in the graph convolution. The key equations of generating the adaptive graph adjacency are shown below:

Where W 1 , W 2 , and W 3 are learnable parameters. E 1 and E 2 are randomly initialized node embeddings that can be learned during training. We represent those embeddings by the spatiotemporal features F ST (e.g., traffic volume) to reveal the underlying dynamic connections between nodes. The subtraction and ReLU activation function in Eq. (6) lead to the asymmetric property of A dynamic . α is a hyperparameter to control the saturation rate. A static is a pre-calculated adjacency matrix before training the model by computing the Pearson correlation coefficient between the risk scores of locations (nodes), where the correlation between node i and node j , a ij = ∑ t ( y i -¯ y i )( y j -¯ y j ) √ ∑ t ( y i -¯ y i ) 2 ( y j -¯ y j ) 2 . The final adjacency matrix A is the weighted sum of A dynamic and A static , and the weight β is learned by a sigmoid activation function in Eq. 7 from the linear transformation of temporal feature F T . Intuitively, the static adjacency matrix indicates a baseline correlation between different locations and it is pre-computed before training. This is also what most of the related work has been done. However, inspired by many observations from related studies [10][40], it is evident that we think this correlation is not always constant. Therefore, we use locations' time-variant features to construct a new adjacency matrix, and this dynamic adjacency matrix varies with time. We learn the parameters in this dynamic matrix during the network training process. Finally, we combine the static and the learned dynamic matrices through a learned weight β . In this way, a combined adjacency matrix can be treated as an adaptation from a static adjacency matrix considering the influence of other features during different periods. Finally, extracted embeddings are fed into LSTM layers and make final predictions of the risk scores y for each location through a fully connected layer.

3.2 Local Ranking Metric and Hybrid Loss function

As previously mentioned, ranking-based metrics such as NDCG are not designed to handle spatial correlations. According to the first law of geography [35], nearby locations may share very similar socio-environmental attributes, thus it is challenging to rank neighboring locations correctly. However, the locations nearby with uncertain event patterns are worthy to be focused on so that more potential events can be discovered. Moreover, using NDCG on event ranking problems can cause overconcentrating on top-ranked locations and sacrificing prediction accuracy on other locations due to lower priority. To address those issues, we design a novel local ranking measurement named Local Normalized Discounted Cumulative Gain (L-NDCG) to measure spatially local ranking quality over every sub-region of the study area. The L-NDCG is calculated as:

We use the same terminologies from Eq. 3. The unique part is that we compute an NDCG score for every location in the study area based on the local ranking of a subset of locations N ( s t ) , where N is a neighborhood of location s t . We define the N as the Euclidean distance of coordinates smaller than R in this work. N can be defined in other ways depending on the need of the problem. Essentially, L-NDCG is the average of NDCG scores for all subsets of locations. In this way, a few stationed hot-spot locations only take considerably large weight in their own NDCG scores and cannot over-influence the overall L-NDCG score. L-NDCG emphasizes ranking correctly on each subset of locations, and a greater L-NDCG score indicates that relatively more important locations can be distinguished from their nearby locations in a small region. Similar to optimizing NDCG, the ranking operator of L-NDCG is non-differentiable, thus we optimize its surrogates instead.

Where g ( w ; x t ˆ s , N ( s t )) is a surrogate loss function similar to Eq. 2, and N ( s t ) is a set of locations. Finally, to learn a trade-off between locally ranking quality and globally ranking quality we design a hybrid objective function consisting of both NDCG and L-NDCG

Where σ is a hyperparameter controlling the preference between NDCG and L-NDCG.

3.3 Importance-based Location Sampling with Spatial Filtering

To bridge the gap between capturing spatial correlations in a list of locations and optimizing the quality of predicted ranking, we propose a novel importance-based location sampling strategy with spatial filtering. Specifically, we first design an importance measure function to assign higher weights to locations with larger errors in predictions and higher ranking priority. Secondly, we sample locations based on the weights assigned by importance-measuring, so that important locations have a higher probability to be sampled. Afterward, only losses from sampled locations will be calculated in the objective function and considered during the optimization. In this way, the model pays attention to more important locations. Thirdly, we adjusted the importance scores every epoch so that the model learns to focus on different locations adaptively. Lastly, spatial filters are applied to smooth the importance scores in each epoch to achieve spatial-aware sampling. This allows nearby locations to be sampled in the same batch with high probability to help the model learn how to rank them. The training process is shown in algorithm 1

Algorithm 1: SpatialRank Training

Algorithm 1 shows the details of the importance-based sampling mechanism. The inputs include feature tensor F , event risk scores y , hyperparameter σ , and Gaussian standard deviation λ . The output is a learned model. The algorithm starts with initializing a set of equal-importance scores, which means the probabilities of locations being sampled are equal in the first epoch. Line 3 is forward propagation with current model parameters. Line 4 calculates the weighted NDCG losses given true label y , predicted labels ˆ y , and current importance scores for locations. Loss NDCG is computed by Eq. 9, where importance scores { p 1 , p 2 , ..., p s } ∈ S are normalized and treated as weights. Line 6 is our novel hybrid loss function discussed in Eq 10 to leverage the local ranking quality. The key step is to update the importance scores set based on current prediction errors and true ranking in Lines 7-8. We design a score function shown in Line 8 to leverage the errors made in predictions and the priority of true ranking for each location.

where r () denotes a ranking function of the i -th location in the study area, and | y t s -ˆ y t s | is the absolute difference between predicted injuries and true injuries. Basically, a higher score E l indicates that

there are larger prediction errors and higher true ranking in this location s . Note that smaller r ( y t s ) denotes a higher true ranking. To capture their geographical connections, we map the list of locations to their original locations on the grids, and then we apply a Gaussian filter to smooth the score distribution spatially in Line 9. λ is the standard deviation. In this way, the model can learn better spatially correlated patterns, and avoid over-focusing on a few standalone locations but ignoring the the area nearby. Next, the smoothed scores E ∗ are normalized into P , where ∑ s P s = 1 . Finally, we compute the gradient ∇ f ( θ ) by loss hybrid and update model parameters θ . After a few iterations, we obtain the learned model f θ .

3.4 Complexity Analysis

In each iteration complexity of SpatialRank, we need to conduct forward propagation h t ( s ; w ) , ∀ s ∈ S + t ∪ S t and back-propagation for computing ∇ h t ( s ; w ) , ∀ s ∈ S + t ∪ S t . The complexity of forwarding is S = ∑ t ∈ T ( |S + t |N ( s ) + |S t | ) d ≤ O ( TSd ) , where d is the model size. To compute local NDCG part, an extra cost on neighbor querying is needed to replace location list size on its complexity. The cost of computing ¯ g ( w ; x , S t ) is ∑ t ∈ T |S + t ||S t | ≤ O ( TS 2 ) for NDCG and ∑ t ∈ T |S + t |N 2 ( s ) ≤ O ( TS N 2 ( s )) for local NDCG. The size of N 2 ( s ) is small, therefore the total cost is reduced to O ( TSd + TS 2 ) .

4 Experiments

We perform comprehensive experiments on three real-world traffic accident and crime datasets from Chicago 2 and the State of Iowa 3 . Experiment results show that our proposed approach substantially outperforms state-of-art baselines on three datasets by up to 12 . 7% , 7 . 8% , and 4 . 2% in NDCG respectively. The case study and results on the state of Iowa are shown in Appendix C.

Data. In the Chicago dataset, we collect data from the year 2019 to the year 2021. The first 18 months of this period are used as the training set, and the last 6 months of 2020 are used as the validating set. The year 2021 is used as a testing set. The area of Chicago is partitioned by 500 m × 500 m square cells and converted to a grid with the size of 64 × 80 . For the crime dataset, we use total crimes as the risk score. For accident datasets, we use the number of injuries as the risk score. w Baselines. First, we use daily Historical Average (HA) . Next, we consider popular machine-learning methods including Long Short-term Memory (LSTM) [16], and Convolutional LSTM Network (ConvLSTM) [31]. Thirdly, we compare with recent methods such as GSNet [37], Hetero-ConvLSTM [40], and HintNet [5]. Moreover, we compared our optimization approach with other NDCG optimization solutions including Cross Entropy (CE) , ApproxNDCG [28], and SONG [29]. The details of the baselines are described in Appendix C.

Metrics. To measure the ranking quality of foremost locations, we test K ∈ [30 , 40 , 50] on the test data. We use the metrics including NDCG, L-NDCG, and top-K precision (Prec)[23]. We report the average performance and standard deviation over 3 runs for three datasets.

4.1 Performance Comparison

In table 1, we can observe that our SpatialRank significantly outperforms other compared baselines in both datasets. We observe a similar trend among all metrics. On both datasets, Hetero-ConvLSTM and HintNet achieve similar results and outperform general machine learning methods such as LSTM and ConvLSTM. GSNet is designed on a dataset with a smaller grid size, thus performing worse on our larger grid. To study the effectiveness of learning the appropriate graph, we set the learning parameter β as a fixed ratio of 0.5 in SpatialRank # . Oppositely, β is a parameter to be learned based on temporal features in SpatialRank. In the Table 1 and Table. 2, each * indicates that the performance improvement of the proposed method over this baseline is statistically significant based on the student t-test with α = 0 . 05 over three runs. The results show that the design of the time-aware graph convolution is able to improve performance and capture more dynamic variations in the graph, therefore performing better over different top-k rankings.

‰: × 10 -3

4.2 Optimization Comparison

To evaluate the effectiveness of our proposed hybrid objective function and importance-based location sampling, we perform experiments on the same network architecture but with different optimization solutions including Cross Entropy (CE), ApproxNDCG, and SONG. The results are shown in table 2. Methods designed to optimize NDCG consistently perform better than Cross Entropy. SpatialRank substantially outperforms SONG and ApproxNDCG and made a noticeable improvement on L-NDCG as it is considered in the objective function.

4.3 Ablation Study

We examine the effects of tuning hyper-parameter σ in the hybrid loss function. We present the results in table 3. Recall that σ decides the ratio of L-NDCG takes in the objective function. σ = 0 means L-NDCG is not considered in the objective function. Starting from σ = 0 , we observe a trend that the overall performance improves steadily while σ goes up, and it reaches the best performance

K = 30

K = 40

K = 50

350 -

300 -

Randomness

SpatialRank

GSNet

F00E

Randomness

SpatialRank

GSNet

-00E

=*=

Randomness

SpatialRank

GSNet

250

HA

HeteroConvLSTM

HintNet

250

HA

HeteroConvLSTM

HintNet

250

HeteroConvLSTM

HA

HintNet

200

200

200

150

150 -

150

100 -

100

100

50 -

50

50

0

0

0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

at σ equals 0.1 in the accident and crime dataset. It indicates that considering a reasonable weight of L-NDCG in the objective function boosts overall performance. These results demonstrate the importance of considering local ranking because NDCG, L-NDCG, and precision can be all improved.

4.4 Cross-K function

We use the Cross-K function[14] with Monte Carlo Simulation[33] to evaluate the accuracy of predicted locations. The Cross-K function measures the spatial correlation between the predicted locations and true locations. Specifically, we calculate the average density of predictions within every distance d of a true event in each day as shown in Eq. 13:

̸

where λ is global density of event j , and I () is an identity function which equals one if real distance d ij is smaller than d , else equals zero. n is the number of events i . The results are shown in Figure 2. The grey curve represents the complete spatial randomness and we use it as a reference baseline. Higher is better. Our SpatialRank achieves the best predictions in both datasets, which indicates that the predictions of SpatialRank are significantly spatially correlated with ground truth. The similar results on the other two datasets are shown in Appendix C in the supplementary materials.

Additional Results and a Case Study to demonstrate successful prediction examples are included in the Supplementary materials.

5 Related Work

Urban Event forecasting has been widely studied in the past few decades. Most early studies [13] [21] [6][19] rely on small-scale and single-source datasets with limited types of features, thus the prediction accuracy is limited. Notably, Zhou et al. [42] proposed a differential time-varying graph convolution network capturing traffic changes and improving prediction accuracy. Similarly, Wang et al. [37] proposed GSNet with a geographical module and a weighted loss function to capture semantic spatial-temporal correlations among regions and solve data sparsity issues. Addressing the issue of spatial heterogeneity, Yuan et al. [40] proposed Hetero-ConvLSTM to leverage an ensemble of predictions from models learned from pre-selected sub-regions. Furthermore, An et al. [5] proposed HintNet partitions the study area hierarchically and transfers learned knowledge over different regions to improve performance. However, most existing models rely on optimizing cross-entropy and the objective is to make accurate predictions on every location. Learning to rank is an extensively studied area in recommendation systems and search engines [22]. NDCG is a widely adopted metric to measure ranking quality. The prominent class of methods involves approximating ranks in NDCG using smooth functions and subsequently optimizing the resultant surrogates. Taylor et al. [34] tries to use rank distributions to smooth NDCG, but suffers from high computational cost. Similarly, Qin et al. [28] approximates the indicator function by a generalized sigmoid function with a top-k variant. Noticeably, Qiu et al. [29] develop stochastic algorithms optimizing the surrogates for NDCG and its top-K variant. Although it is possible to directly apply the existing ranking methods to the urban event ranking problem, the performance tends to be unsatisfactory as these methods commonly assume independence between items and queries and lack the ability to handle spatiotemporal autocorrelation in the data.

6 Conclusion and Limitations

In this work, we formulate event forecasting as a location ranking problem. We propose a novel method SpatialRank to learn from spatiotemporal data by optimizing NDCG surrogates. To capture dynamic spatial correlations, we design an adaptive graph convolution layer to learn the graph from features. Furthermore, we propose a hybrid loss function to capture potential risks around hot-spot regions, and a novel ranking-based importance sampling mechanism to leverage the importance of each location considered during the model training. Extensive experimental results on three real-world datasets demonstrate the superiority of SpatialRank compared to baseline methods.

Limitations : the model's performance might be affected by other properties of data, such as spatial heterogeneity and sparsity. We observe less improvement over baselines on the Iowa dataset, partially due to that the data is sparser over a large area with heterogeneity. These are issues addressed by some of the prior work and can be addressed in our future work. In addition, the new algorithm increases the training time complexity due to the sampling steps. This is acceptable due to the relatively small number of locations and time periods in urban event datasets but may require extra work to generalize to large datasets.

Acknowledgments and Disclosure of Funding

T. Yang was partially supported by NSF Career Award 18844403, NSF Fair AI Grant 2147253, NSF RI Grant 2110545.

References

Chicago Police Department, https://home.chicagopolice.org/cpd-expands-smart-policing-technology-tosupport-strategic-deployment-and-cta-safety/ . 2020.

Fox News, https://www.foxnews.com/us/us-murder-rate-continued-grim-climb-i2021-new-fbi-estimatesshow . 2022.

Police Executive Research Forum, https://www.policeforum.org/workforcemarch2022 . 2022.

J. Abellán, G. López, and J. De Oña. Analysis of traffic accident severity using decision rules via decision trees. Expert Systems with Applications , 40(15):6047-6054, 2013.

B. An, A. Vahedian, X. Zhou, W. N. Street, and Y. Li. Hintnet: Hierarchical knowledge transfer networks for traffic accident forecasting on heterogeneous spatio-temporal data. Proceedings of the 2022 SIAM International Conference on Data Mining , 2022.

L. Barba, N. Rodríguez, and C. Montt. Smoothing strategies combined with arima and neural networks to improve the forecasting of traffic accidents. The Scientific World Journal , 2014, 2014.

R. Bergel-Hayat, M. Debbarh, C. Antoniou, and G. Yannis. Explaining the road accident risk: Weather effects. Accident Analysis & Prevention , 60:456-465, 2013.

K. Bhatia, H. Jain, P. Kar, M. Varma, and P. Jain. Sparse local embeddings for extreme multi-label classification. In Advances in Neural Information Processing Systems , volume 29, pages 730-738, 2015.

C. Caliendo, M. Guida, and A. Parisi. A crash-prediction model for multilane roads. Accident Analysis & Prevention , 39(4):657-670, 2007.

R. N. Carey and K. M. Sarma. Impact of daylight saving time on road traffic collision risk: a systematic review. BMJ Open , 7(6), 2017.

L.-Y. Chang. Analysis of freeway accident frequencies: negative binomial regression versus artificial neural network. Safety science , 43(8):541-557, 2005.

L.-Y. Chang and W.-C. Chen. Data mining of tree-based models to analyze freeway accident frequency. Journal of safety research , 36(4):365-375, 2005.

M. M. Chong, A. Abraham, and M. Paprzycki. Traffic accident analysis using decision trees and neural networks. ArXiv , cs.AI/0405050, 2004.

P. M. Dixon. Ripley's K Function . John Wiley & Sons, Ltd, 2014.

N. Dong, H. Huang, and L. Zheng. Support vector machine in crash prediction at the level of traffic analysis zones: Assessing the spatial proximity effects. Accident Analysis & Prevention .

S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation , 9(8):1735-1780, 1997.

K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems , 20(4):422-446, 2002.

A. Khezerlou, X. Zhou, L. Li, Z. Shafiq, A. Liu, and F. Zhang. A traffic flow approach to early detection of gathering events: Comprehensive results. ACM transactions on intelligent systems and technology , 8(6):1-24, 2017.

A. V. Khezerlou, X. Zhou, X. Li, W. N. Street, and Y. Li. Dilsa+: Predicting urban dispersal events through deep survival analysis with enhanced urban features. ACM transactions on intelligent systems and technology , 12(4):1-25, 2021.

Z. Li, C. Huang, L. Xia, Y. Xu, and J. Pei. Spatial-temporal hypergraph self-supervised learning for crime prediction. In arXiv.org , Ithaca, 2022. Cornell University Library, arXiv.org.

L. Lin, Q. Wang, and A. W. Sadek. A novel variable selection method based on frequent pattern tree for real-time traffic accident risk prediction. Transportation Research Part C , 55, 2015.

T.-Y. Liu. Learning to Rank for Information Retrieval. Springer, 2011.

J. Lu, C. Xu, W. Zhang, L. Duan, and T. Mei. Sampling wisely: Deep image embedding by top-k precision optimization. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 7960-7969. IEEE, 2019.

X. Mao, C. Yuan, J. Gan, and S. Zhang. Risk factors affecting traffic accidents at urban weaving sections: Evidence from china. International journal of environmental research and public health , 16(9):1542, 2019.

T. R. Miller, M. A. Cohen, D. I. Swedler, B. Ali, and D. V. Hendrie. Incidence and costs of personal and property crimes in the usa, 2017. Journal of benefit-cost analysis , 12(1):24-54, 2021.

A. Najjar, S. Kaneko, and Y. Miyanaga. Combining satellite imagery and open data to map road safety. In Thirty-First AAAI Conference on Artificial Intelligence , 2017.

NHTSA. 2021 fatality data show increased traffic fatalities during pandemic. https://www.nhtsa.gov/ press-releases/early-estimate-2021-traffic-fatalities , Jun 2021.

T. Qin, T.-Y. Liu, and H. Li. A general approximation framework for direct optimization of information retrieval measures. Information Retrieval , 13(4):375-397, 2010.

Z. Qiu, Q. Hu, Y. Zhong, L. Zhang, and T. Yang. Large-scale stochastic optimization of ndcg surrogates for deep learning with provable convergence. In International Conference on Machine Learning , 2022.

H. Ren, Y. Song, J. Wang, Y. Hu, and J. Lei. A deep learning approach to the citywide traffic accident risk prediction. 2018 21st International Conference on Intelligent Transportation Systems (ITSC) , pages 3346-3351, 2018.

X. Shi, Z. Chen, H. Wang, D.-Y. Yeung, W.-k. Wong, and W.-c. Woo. Convolutional lstm network: A machine learning approach for precipitation nowcasting. 2015.

R. Swezey, A. Grover, B. Charron, and S. Ermon. Pirank: Scalable learning to rank via differentiable sorting. Advances in Neural Information Processing Systems , 34, 2021.

R. Tao and J.-C. Thill. Flow cross k-function: a bivariate flow analytical method. International journal of geographical information science : IJGIS , 33(10):2055-2071, 2019.

M. Taylor, J. Guiver, S. Robertson, and T. Minka. Softrank: optimizing non-smooth rank metrics. In Proceedings of the 2008 International Conference on Web Search and Web Data Mining , pages 77-86, 2008.

W. R. Tobler. A computer movie simulating urban growth in the detroit region. Economic geography , 46:234-240, 1970.

A. Vahedian, X. Zhou, L. Tong, W. N. Street, and Y. Li. Predicting urban dispersal events: A two-stage framework through deep survival analysis on mobility data. 2019.

B. Wang, Y. Lin, S. Guo, and H. Wan. Gsnet: Learning spatial-temporal correlations from geographical and semantic aspects for traffic accident risk forecasting. In 2021 AAAI Conference on Artificial Intelligence (AAAI'21) , 2021.

M. Wu, Y. Chang, Z. Zheng, and H. Zha. Smoothing dcg for learning to rank: a novel approach using smoothed hinge functions. In Proceedings of the 18th ACM conference on information and knowledge management , CIKM '09, pages 1923-1926. ACM, 2009.

Z. Wu, S. Pan, G. Long, J. Jiang, X. Chang, and C. Zhang. Connecting the dots: Multivariate time series forecasting with graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining , pages 753-763, 2020.

Z. Yuan, X. Zhou, and T. Yang. Hetero-convlstm: A deep learning approach to traffic accident prediction on heterogeneous spatio-temporal data. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 984-992, 2018.

Y. Zhang, Y. Li, X. Zhou, X. Kong, and J. Luo. Trafficgan: Off-deployment traffic estimation with traffic generative adversarial networks. 2019 IEEE International Conference on Data Mining (ICDM) , pages 1474-1479, 2019.

Z. Zhou, Y. Wang, X. Xie, L. Chen, and C. Zhu. Foresee urban sparse traffic accidents: A spatiotemporal multi-granularity perspective. IEEE Transactions on Knowledge and Data Engineering , pages 1-1, 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper casts event forecasting as location ranking problem. They propose a spatial event ranking approach called SpatialRank. SpatialRank optimizes the NDCG metric while taking spatiotemporal autocorrelation into account. \n\nSpatialrank uses a graph convolutional network to encode autocorrelated input and applies NDCG optimization on the top fully-connected layer. Based on the experiments on 3 datasets, it outperforms the underlying approaches on precision@k metric.\n\nSTRENGTHS:\nThis paper is a good application of graph convolutional networks to autocorrelated spatiotemporal data to solve the event ranking problem. None of the individual pieces are unique, but the compound technique is a novel application.\n\n* Introduction of L-NDCG is novel as it addresses the problem of hotspots dominating the overall NDCG metric.\n* It is also nice that the authors performed an ablation study -- albeit being rather minimal.\n* Ideas are fairly clear and explained well.\n* The work might be significant if its applicability, which is teased in Section 6, is discussed in more detail.\n\nWEAKNESSES:\nA major weakness of the paper is its presentation. It needs attention to detail as well as how it is presented overall before being ready to be published. Some of the presentation issues are as follows:\n* Lines 41-61 is basically a Related Work section. However, the reader is greeted with an actual Related Work section on the same page where most of the information is repeated.\n* Line 151. What's Adam-style optimizer? Do you mean to say gradient-based optimizers?\n* The main contribution of the paper is illustrated in Figure 1 of the Appendix. It deserves to be in the actual paper.\n* Iowa dataset needs to be in the paper and the existing tables should be squeezed as they don't carry too much information to deserve that much space.\n* The paper occasionally leads the reader to assume that the actual task is to increase the NDCG score although the problem is to predict top k events.\n* A few spelling, capitalization, or other errors:\n  * Line 86: Related Works -> Related Work\n  * Line 182: Relu -> ReLU\n  * Algorithm 1: Normorlize -> normalize\n  * Line 280: We -> we\n  * Line 281: Convlstm -> ConvLSTM\n  * K@30 -- can't really tell this means. It is maybe K=30?\n\nOne other main weakness of the paper is its over-reliance on NDCG. First of all, experiments should't compare the approaches based on NDCG as it is the metric this approach is optimizing for as it wouldn't make a fair comparison per Goodhart’s law. Secondly, it is stated that \"NDCG is not a perfect metric for our problems as it neither measures the local ranking quality nor considers spatial autocorrelation among locations.\" However, NDCG is still treated as somewhat ground truth throughout the paper.\n\nOnce the problem is converted into NDCG optimization, other approaches can also be utilized. For the experiments, tree-based ranking models can also be evaluated to replace the final FC layer as they have the potential to perform better.\n\nQUESTIONS:\n* What are the quantitative measures of performance wrt baselines?\n* Do subregions need to be given?\n\nLIMITATIONS:\nAuthors have addressed a few limitations including performance, data sparsity. To stress-test the approach the authors are advised to test their method on more variety of datasets. One can't confidently enumerate all limitations before it is tested on many datasets.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method which applies learning to rank losses to spatiotemporal event prediction. The observed data is considered over a discrete spatial partitioning and considers a time series of feature information. Features are grouped into purely temporal, purely spatial (not-time dependent) and spatiotemporal information. In addition, there is a risk score >0 for time, location pairs where an event occurred. The examined task is now to predict the top-k riskiest location for the next time step.\nThe method encodes the state of the network using a know spatiotemporal prediction network which employs graph NN layers to encode the underlying spatial connectedness modelling a road network. \nThe paper's novelty is to learn a dynamic adjacency matrix from the spatiotemporal feature of each location, which allows us to dynamically model the current conditions, such as traffic.  This dynamic adjacency matrix is combined with the static one.\nA second novelty is adapting the ranking score to consider local rankings, which replace the global rank of location the original NDCG with a ranking score within a local neighbourhood. The localized NDCG is also combined with global NDCG loss.\nFot training, the authors propose to compute a weight for each element which is based on the ranking score of the  prediction error.\n\nSTRENGTHS:\nThe paper proposes a novel idea to compute a ranking of locations instead of directly computing an event probability or even the next event, which might be hard to measure. In other words, instead of predicting the event likelihood, a ranking is learnt, which should be easier to learn but sufficient for various tasks.\n\nThough the general layout of the proposed method is to combine NDCG with a spatial prediction backbone, the paper adds three modifications and shows that they considerably improve the performance compared to baselines.\n\nThe paper is good to follow and provides a reproducible description of the proposed methods.\n\nThe experiments show improved results von two real-world data sets, and the authors provide an ablation study.\n\nThe paper provided sufficient supplementary material to check all the details.\n\nWEAKNESSES:\nThough I could follow the reasoning of computing a top-k query, the motivation why this information is enough in several applications could be motivated better in the introduction.\n\nThe ablation study seems only to compare various values of \\delta. But it would also be interesting to see the impact of the other contributions. As the technical contribution seems to consist of three relatively independent adaptions to the spatiotemporal setting seeing how much each contribution added to the improvement might be very insightful. \n\nThe description of the dynamic adjacency matrix is a little bit cryptic. In particular, it is unclear how it is trained and why it should represent a dynamic adaption of the static adjacency matrix\n\nQUESTIONS:\nWhat is the intuition of subnetwork for the dynamic adjacency matrix, and what is it supposed to model? Can you give more details about the intuition of the architecture and how it naturally aligns with the static adjacency matrix? \n(There should not be a link if there is no connection in the static matrix, right?).\n\nCan you give details on how much the instance weighting in training helped to improve the performance or speed up the conference? \n\nSimilarly, is there an ablation study on the dynamic adjacency matrix?\n\nLIMITATIONS:\nThe paper addresses its limitations in a dedicated paragraph.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a deep learning model called SpatialRank that predicts the top-k riskiest locations of future events such as traffic accidents and crimes by optimizing a spatial version of the NDCG measure. The model features adaptive graph convolution layers that learn the spatiotemporal dependencies from data, a hybrid loss function that balances global and local ranking quality, and an importance-sampling with spatial filtering algorithm that guides the model to focus on important locations. The model is evaluated on three real-world datasets from Chicago and Iowa, and outperforms several methods in terms of NDCG, L-NDCG, and precision. The model also demonstrates better spatial correlation with ground truth using the cross-K function.\n\nSTRENGTHS:\nS1 The proposed method is evaluated on three different real-life datasets.\n\nS2 The idea of introducing the NDCG metric into a location-based learning problem is somewhat interesting.\n\nWEAKNESSES:\nThough this paper arises an interesting problem, I have the following major concerns about this paper:\n\n(1) The motivation for urban event ranking is weak. The paper does not highlight the significance and novelty of this problem, and why it is more important than making event predictions for each location. The paper should provide more evidence or examples to motivate the need and value of ranking locations for future events.\n\n (2) The technical contribution is limited and the model is not novel enough. The paper does not clearly state how the proposed model differs from existing methods in terms of problem formulation, model design, optimization strategy, or evaluation metric. The designed NDCG loss function is quite similar to Eq. (3). I do not think there is much difference between these two functions. The paper should provide more details and analysis to demonstrate the advantages and challenges of the proposed approach.\n\n(3) The model uses a Euclidean distance to define the neighborhood of each location, which may not reflect the actual spatial proximity or connectivity of locations in terms of road network or travel distance.\n\n(4) The presentation and description of the framework is poor and incomplete. There are several unclear or confusing points in the paper, please refer to the above Questions.\n\n(5) The experiments are not sufficient. There are several limitations or missing details in the experimental section, such as:\n(5.1) No significance test to show whether the differences between SpatialRank and baselines are statistically significant or not.\n(5.2) No ablation study experiments to show the effectiveness or necessity of each component of SpatialRank, such as the dynamic graph generation module or the designed NDCG loss function.\n\n- Minor concerns: \n\n(6) There are some grammar issues in the paper, such as “learning-to-ranking” in Line 51, which should be corrected.\n\n(7) The paper lacks a framework figure to illustrate the overall architecture and workflow of the model, which makes it hard to follow and understand.\n\nQUESTIONS:\nQ1. Directly constructing from temporal events is more effective and explicit. Why generating a dynamic graph from features is necessary or effective? How does it capture the dynamic spatiotemporal dependencies among locations? How does it compare with directly constructing a graph from temporal events?\n\nQ2. Why the method set E1 = E2 = FST in Eq. (4) and Eq. (5)? Since these two equations are almost same, why combine the two embedding Z_1 and Z_2 in Eq. (6) for generating the dynamic adjacency matrix?\n\nLIMITATIONS:\nThe authors have discussed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the problem of future urban event prediction on spatiotemporal data. This is an important problem for a broad range of urban application. Different from prior work, this paper for the first time predicts most likely future events by directly optimizing location ranking in the prediction through NDCG optimization. The authors propose a dynamic adjacency matrix of locations, a hybrid NDCG loss function and a spatial sampling algorithm to handle the unique challenges brought by spatiotemporal data. Experimental results on three datasets show that the proposed solution can beat state-of-the-art event prediction models as well as existing NDCG optimization solutions on crime and accident prediction.\n\nSTRENGTHS:\n1.\tThe problem investigated by the paper is significant for many urban applications such as crime prediction and accident forecasting. \n2.\tThe paper is the first to use NDCG optimization on spatiotemporal data for location ranking. This is a novel idea for event prediction.\n3.\tThe work also adds value to the literature of ranking algorithm for providing solutions on how to handle non-iid spatial data ranking. \n4.\tThe experiments show that the proposed SpatialRank model outperforms not only event/accident prediction methods but also traditional NDCG optimization method. This suggests that the model is effective in addressing some of the unique challenges in spatiotemporal data.\n\nWEAKNESSES:\n1.\tAs discussed by the authors, the hybrid loss function might introduce significant computation cost increase to the learning algorithm. In addition to the complexity analysis, the authors should provide additional evidence (e.g., experiments) to justify the impact to training time.\n\n2.\tThere are quite a lot of symbols in the paper and the authors should provide a table or summary of these symbols. Without such information, the complexity analysis part is a bit hard to follow.\n\n3.\tThe experimental results should be presented with error bars.\n\nQUESTIONS:\n1.\tWhat is the model architecture used for the optimization comparison experiment? How does the model architecture choice affect the performance of the optimizers? Is it possible that with a different network architecture the other optimizers such as SONG or approxNDCG can beat spatialRank?\n2.\tHow is top-k precision defined in the experiments?\n\nLIMITATIONS:\nN/A.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}