SpatialRank: Urban Event Ranking with NDCG Optimization on Spatiotemporal Data
Authors: BANG AN
Xun Zhou
Yongjian Zhong
Tianbao Yang
Venue: NeurIPS 2023 poster (2023)
Keywords: urban event
NDCG optimization
ranking
traffic accident
crime
spatiotemporal data
TL;DR: We propose the first spatial NDCG optimization method for urban event location ranking on spatiotemporal data

Abstract: The problem of urban event ranking aims at predicting the top-$k$ most risky locations of future events such as traffic accidents and crimes. This problem is of fundamental importance to public safety and urban administration especially when limited resources are available. The problem is, however, challenging due to complex and dynamic spatio-temporal correlations between locations, uneven distribution of urban events in space, and the difficulty to correctly rank nearby locations with similar features. Prior works on event forecasting mostly aim at accurately predicting the actual risk score or counts of events for all the locations. Rankings obtained as such usually have low quality due to prediction errors. Learning-to-rank methods directly optimize measures such as Normalized Discounted Cumulative Gain (NDCG), but cannot handle the spatiotemporal autocorrelation existing among locations. Due to the common assumption that items are independent. In this paper, we bridge the gap by proposing a novel spatial event ranking approach named SpatialRank. SpatialRank features  adaptive graph convolution layers that dynamically learn the spatiotemporal dependencies across locations from data. In addition, the model optimizes through surrogates a hybrid NDCG loss with a spatial component to better rank neighboring spatial locations. We design an importance-sampling with a spatial filtering algorithm to effectively evaluate the loss during training. Comprehensive experiments on three real-world datasets demonstrate that SpatialRank can effectively identify the top riskiest locations of crimes and traffic accidents and outperform state-of-art methods in terms of NDCG by up to 12.7%.

SpatialRank: Urban Event Ranking with NDCG Optimization on Spatiotemporal Data
Bang An
Xun Zhou
Department of Business Analytics University of Iowa lowa City, IA 52242
Department of Business Analytics
University of lowa Iowa City, IA 52242
edu
xun-zhouquiowa. edu
Yongjian Zhong
Tianbao Yang
Department of Computer Science
Department of Computer Science and Engineering
University Iowa lowa City, IA 52242 yonggiann-2ongguuiora edu
Texas A&M University
College Station, TX 77843
Lunbao-yang@tam edu
Abstract
The problem of urban event ranking aims predicting the top-k most risky lo- cations of future events such as traffic accidents and crimes. This problem is fundamental importance to public safety and urban administration especially when limited resources are available. The problem is, however, challenging due to complex and dynamic spatio-temporal correlations between locations, uneven distri- bution of urban events in space, and the difficulty to correctly rank nearby locations with similar features. Prior works on event forecasting mostly aim accurately predicting the actual risk score or counts of events for all the locations. Rankings obtained as such usually have low quality due to prediction errors. Learning-to- -t-rrnkkk methods directly optimize measures such as Normalized Discounted Cumulative Gain (NDCG), but cannot handle the spatiotemporal autocorrelation existing among locations. In this paper, we bridge the gap by proposing novel spatial event rank- ing approach named SaatialRank. SpatialRank features adaptive graph convolution layers that dynamically learn the spatiotemporal dependencies across locations from data. In addition, the model optimizes through surrogates a hybrid NDCG loss with spatial component to better rank neighboring spatial locations. We design an importance-sampa sampling with a spatial filtering algorithm to effectively evaluate the loss during training. Comprehensive experiments three real-world datasets demonstrate that SpatialRank can effectively identify the top riskiest locations crimes and traffic accidents and outperform state-of-the-ar methods in terms of NDCG by up to 12.7%.
Introduction
Given risk score defined based historical urban events (e.g., crimes, traffic accidents) in study area as well as the prommentilo attributes (e.g., travel demands, weather, road network) associated with the events, the goal of the urban event ranking problem is to learn model that can predict the ranking of the top-k riskiest locations of future events.
"corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

The urban event ranking problem is a widely observed spatiotemporal prediction problem, which has critical upplications in public safety, traffic management, and urban planning. For example, the National Highway Traffic Safety Administration [7 estimated a total of 42,939 deaths in motor vehicle traffic crashes in the United States in 2021. Crime control and property loss cost over $2.5 trillion in the United States in 2017 [25]. Chicago Police Department has utilized criminal intelligence analysis and data science techniques to help command staff determine where best to deploy resources D. However, according to the Police Executive Research Forum, there were 42. 72. more resignations among law enforcement but a 3.9% decrease in hiring new officers in 2021 compared to 2019 [3]. Meanwhile, the Federal Bureau of Investigation confirms that violent crime in 2020 has surged nearly 30% over 2019 [2). Therefore, given such growth of crimes and accidents, predicting the riskiest locations of traffic accidents crimes helps law enforcement stakeholders allocate their limited resources strategically to reduce injuries, deaths, and property losses.
Despite its importance, the urban event ranking problem is technically challenging. First, there exist complex and dynamic spatiotemporal correlations between locations in terms of events and attributes such traffic conditions and weather changes. Capturing such dynamic relationships is non-trivial. Second, due to the presence of spatial autocorrelation, nearby locations may share very similar socio-environmental attributes. It is thus very challenging to learn good model that can correctly rank neighboring locations. Finally, urban events are usually sparse in space. Making a prediction with low average error in count does not ensure good ranking of the top-k locations.
Prior works on urban event prediction employed either classic machine learning methods 18 or deep learning techniques including recurrent neural networks 30 and convolutional neural networks [26] (36.. Recently, Li et al. (0) proposed cross-region hypergraph structure network to address the data sparsity issues. Besides, Yuan et al. 44) [S] proposed spatial ensembles to address spatial heterogeneity. The objective of the above methods is to make accurate predictions of the risk or count for all locations. However, urban events are often very sparse in space, which might guide the model to avoid predicting accidents most locations to obtain a low average error. Such prediction does not truly benefit the users such as police officers. The top-k locations derived from such predictions are naturally inaccurate.
15
Our problem is also relevant to the learning-tor -o-rinnk problem frequently studied in the field of recommendation systems. State-of-the-arr methods in this area typically predict the top-k items most likely to be chosen by users through optimizing ranking-based metrics such as the Normalized Discounted Cumulative Gain (NDCG) [ The rank operator of NDCG non-differentiable, and previous studies have made noticeable progress in approximating NDCG by surrogate functions 332 (28 (34 [29. In our problem, a straightforward adaptation would be to consider the locations as "items" and each time slot as a "user". However, NDCG is not perfect objective function for our problems as it neither measures the local ranking quality nor considers spatial autocorrelation among locations. Instead, the existing approaches assume items independent. Therefore, directly applying existing NDCG optimization solutions might not be the best solution to our problem.
In this paper, we bridge the gaps in both fields by formulating the urban event ranking problem as a spatial learning-to-rank problem and solving by directly optimizing a "spatial" version of the NDCG measure. We propose SpatialRank, our deep learning model with three novel designs. To efficiently capture the spatial and temporal dependencies. we design an adaptive graph convolution layer that learns the correlations between locations dynamically from features and historical events patterns; to ensure the model balances both the global ranking quality on all the locations and the local ranking quality on subsets of locations, we propose a hybrid loss function combining both NDCG loss and a novel local NDCG loss; to improve the effectiveness of optimizing NDCG surrogates in the spatiotemporal setting of our problem, we design an importance-basst spatial filtering algorithm to iteratively adjust the weights of each location considered in the objective function, thereby guiding the model to concentrate on learning for more important locations. We conduct comprehensive experiments on three real-world datasets collected from Chicago and the state of lowa. The results demonstrate that SpatialRank can substantially outperform baselines and achieve better ranking quality.
location sampling with
Our contributions are summarized below:
To the best of our knowledge, this is the first paper to formulate urban event forecasting as location ranking problem and learn a ranking model by optimizing its ranking quality.
We propose SpatialRank with adaptive graph convolution layers learning from historical event patterns and spatiotemporal features to capture dynamic correlations over time and space.

---

We propose a hybrid objective function to leverage the trade-off between global ranking quality and local ranking quality.
We propose ranking-base based importance sampling algorithm to adaptively adjust the weights of different locations considered in the objective function of the prediction results from the last training epoch to help the model focus on important locations.
Preliminaries
2.1 Formulation of the event ranking problem
spatio-temporal field 8 T is three-dimensional partitioned space, where T' (2,,,, (T} is a study period divided into equal-length intervals (e.g., hours, days) and 8 = { ((M,N) is M x N two-dimension spatial grid partitioned from the study area. set of aocio-envirommenall features F are observed over S x T. which include temporal features F; E IR" (e.g., day of week, holiday), spatial features F, R (e.g., total road length, speed limit), and spatiotemporal features Fat R. x (e.g., traffic volume, rainfall amount). A risk score E IR x is user-defined attribute over T measuring the risk level of each spatiotemporal location (e-g.. number of crimes, injuries of traffic accidents). y(s,t) >> O when any events occurred in (s, L), and equals when no events occurred.
Given the socio-environmental features F' and risk score for all the locations in time window (1,, f., ..; }. the urban event ranking problem to predict the ranking of the top-k locations
fsi...., } S in the next time interval fn+1 with the highest risk scores. The objective is to prioritize the ranking quality on the top-k riskiest locations. As a basic assumption of our problem. there exists spatial and temporal autocorrelation among locations in F, meaning nearby locations tend to have more correlated values. In addition, events are sparse so y is for the majority of the locations. A detailed example of data attributes and feature generation steps can be found in the supplementary materials Appendix A.
2.2 Stochastic Optimization of NDCG in Event Ranking Problem
In the field of recommendation systems, learning to rank is substantially studied, and NDCG is widely used the metric to measure the ranking quality of the foremost importance. In the following part, we use the terminologies from the event forecasting problem to define NDCG in our problem setting. For ranked list of locations s S in period T, the NDCG score computed as by:
1 NDCG, Z, SES,
Σ
20
log, + r(s)
where is the risk score of the location r(s) denotes the rank of location s in the studied spatial domain S,, and Z; the Discounted Cumulative Gain (DCG) score [17] of the perfect ranking of
locations for time period f. However, the rank operator in NDCG is non-differentiable in terms of model parameters, and thus cannot be optimized directly. A popular solution is to approximate rank operator with smooth functions and then optimize its surrogates 34
28
29
as shown in Eq. 2
d(w:x,S.) = Σ E(hi(s': (s:
(2)
s'ES,
where rank operator r(s) in NDCG is approximated by a differentiable surrogate function E(), and the squared hinge loss {(x) max(0, c)² is commonly used [38]. In this way, the model parameters w can be updated by gradient- gradient-based optimizer. We can maximize over L(w)):
2
max L(w)= WERd
T
Z log, w;
SEST

---

time
Dosse bus
Fly
Fiz
Fs
Yes1
GC
GC
NDCG
FC
+
Hybvid Loss
Shoet TTrm Memory
8++1
Local NDOG
Layer
Figure 1: RpatialRank Architecture. The spatiotemporal features are used to generate adjacency matrices and then embedded by graph convolution layers. We use fully connected layer to make final predictions. The hybrid objective function is combined with NDCG loss and local NDCG loss.
where s E S denotes a set of locations with positive risk scores to be considered in the objective function. In this way, the optimization solution NDCG can be directly applied the ranking problem.
Methodology
Directly optimizing NDCG as described above might provide a solution to our problem but fails capture the spatiotemporal autocorrelation in the data. Also, since NDCG global measure, the model might not be able to learn how to rank nearby locations with highly correlated features correctly. In this section, we present our SpatialRank method address these limitations. Figure. demonstrates the proposed model architecture.
3.1 The Deep Learning Model
Many recent deep learning models for spatiotemporal could used as the backbone architecture of our SpatialRank method. A key idea in these methods is to model the correlations between locations using graph, with locations as nodes and edge weights representing the strength correlations, and extract latent spatial dependency information through graph convolution layers 37 [ We follow similar idea build the overall network architecture, where spatial and spatiotemporal features are fed into graph convolutional layers. Then the extracted latent representations are concatenated with temporal features and fed into L.TT layers to capture temporal representations before the final output layer.
key novelty in our SpatialRank network is the design of a time-guided and traffic-aware graph- generating process for the graph convolution layers. Prior works typically use Pearson's 's correlation coefficients similar measures features (e.g., traffic volume, accident counts) between locations as their correlation strengths and pre-compute time invariant adjacency matrix of locations [41] In fact, studies [00 (24 demonstrate that the influence of traffic conditions on events varies over different periods. Inspired by related work [39 on a different problem, we use historical events to generate a static graph and learn a time-variant graph from EST (e.g., traffic volume) each time interval. Intuitively, we use Fr (e.g. hour of the day) to learn the weights of dynamic graph vs. static graph be considered the graph convolution. The key equations generating the adaptive graph adjacency are shown below:

---

Z1 = tanh( W1)
(4)
Z2 tanh(o:E,W:
(5)
Adynamic = =ReLU(tanh(a(2223 2,
(6)
B wigmoid(F+Waa
(7)
A = B.Aymmmmic + (1 3).
(8)
Where W1, W2, and W3 are learnable parameters. E1 and E2 are randomly initialized node embed- dings that can be learned during training. We represent those embeddings by the spatiotemporal features FST (e.g., traffic volume) to reveal the underlying dynamic connections between nodes. The subtraction and ReLU activation function in Eq. (6) lead to the asymmetric property of Adynamic- a a hyperparameter to control the saturation rate. Astatic is a pre-calculated adjacency matrix before training the model by computing the Pearson correlation coefficient between the risk scores of locations (nodes), where the correlation between node and node j. (ej = E.( (1, -1,)))
The final adjacency matrix A is the weighted sum of Adynamic and Astaticr and the weight /3 is learned by sigmoid activation function Eq. from the linear transformation of temporal feature Fr. Intuitively, the static adjacency matrix indicates baseline correlation between different locations and it is pre-computed before training. This also what most of the related work has been done. However, inspired by many observations from related studies [10] 40], it is evident that we think this correlation is not always constant. Therefore, we use locations' time-variant features to construct new adjacency matrix, and this dynamic adjacency matrix varies with time. We learn the parameters this dynamic matrix during the network training process. Finally, we combine the static and the learned dynamic matrices through learned weight B. In this way, combined adjacency matrix can be treated as an adaptation from a static adjacency matrix considering the influence of other features during different periods. Finally, extracted embeddings are fed into LSTM layers and make final predictions of the risk scores y for each location through a fully connected layer.
VE,(u (n. -9,1²²
3.2 Local Ranking Metric and Hybrid Loss function
As previously mentioned, ranking- based metrics such NDCG are not designed to handle spatial correlations. According to the first law of geography 35). nearby locations may share very similar socio-environmentae attributes, thus it is challenging to rank neighboring locations correctly. However, the locations nearby with uncertain event patterns are worthy to be focused on so that more potential events can be discovered. Moreover, using NDCG on event ranking problems can cause over- concentrating on top-ranked locations and sacrificing prediction accuracy on other locations due to lower priority. To address those issues, design novel local ranking measurement named Local Normalized Discounted Cumulative Gain (L-NDCG) to measure spatially local ranking quality over every sub-region of the study area. The L-NDCG is calculated as:
T Σ
2
L-NDCG TIS [=1 s'ES, BENT(a)) ZN(cee (rg,(r( +1)
E
We use the same terminologies from Eq. 3 The unique part is that we compute NDCG score for every location in the study area based on the local ranking of subset of locations (V(s),, where A is neighborhood of location s². We define the A as the Euclidean distance of coordinates smaller than in this work. A can be defined in other ways depending on the need the problem. Essentially, L-NDCG is the average of NDCG scores for all subsets of locations. In this way, few stationed hot- locations only take considerably large weight in their own NDCG scores and cannot over-influence the overall L-NDCGG score. L-NDCG emphasizes ranking correctly each subset of locations, and greater L.-DDCG score indicates that relatively more important locations be distinguished from their nearby locations in small region. Similar optimizing NDCG, the ranking operator of L-NDCG is non-difierentiable, thus we optimize its surrogates instead.
2 -1
max L(w)= WERd
Σ
E
Σ
Σ
(10)
T
s'est SEA((a)) IN(oe 1=1
5

---

Where s(, (s*)) is surrogate loss function similar to Eq. 2 and (ss'') is a set of locations. Finally, to learn a trade-off between locally ranking quality and globally ranking quality we design hybrid objective function consisting of both NDCG and L--DDGG
Loss = (1 oDDD + oL-NDCG
(11)
Where a a hyperparameter controlling the preference between NDCG and L-NDCG.
3.3 Importance-based Location Sampling with Spatial Filtering
To bridge the gap between capturing spatial correlations in a list of locations and optimizing the quality of predicted ranking, we propose novel importance-based location sampling strategy with spatial filtering. Specifically, we first design an importance measure function to assign higher weights to locations with larger errors in predictions and higher ranking priority. Secondly, we sample so that important locations have a
locations based on the weights assigned by importance-measuringg higher probability to be sampled. Afterward, only losses from sampled locations will be calculated in the objective function and considered during the optimization. In this way, the model pays attention more important locations. Thirdly, we adjusted the importance scores every epoch so that the model learns to focus on different locations adaptively. Lastly, spatial filters are applied to smooth
the importance scores in each epoch achieve spatial-aware sampling. This allows nearby locations to be sampled in the same batch with high probability to help the model learn how to rank them. The training process is shown in algorithm
Algorithm 1: SpatialRamn Rank Training
Input: feature tensor F, event risk scores y. hyperparameter d, standard deviation À Output: learned model fe
Initialize probability set = (p1. Pp..,P....,PP L;
2 for each epoch do Compute y = (ar)
(y, Compute loss DCG = WeightedLosst Loss(y, Ú, P)
Compute lo88tocal LocalWeightedl
P)
losShybrii (1 - σ) loSSNDCG + losStocol for s
S do E,
1 T E. 1
(og2(1+r(gg))
E = Σ T for s E E x (6)
10
Update P normorlize(E. [E)
11
Compute gradient (() by losShybrid Update 0 by (()
12
return fo
Algorithm shows the details of the importance-basee based sampling mechanism. The inputs include feature tensor F. event risk scores y. hyperparameter ar, and Gaussian standard deviation X. The output is learned model. The algorithm starts with initializing a set of equal-importance scores, which means the probabilities of locations being sampled are equal in the first epoch. Line 3 is forward propagation with current model parameters. Line calculates the weighted NDCG losses given true label y. predicted labels ÿ. and current importance scores for locations. Loss DOG is computed by Eq. 9 where importance scores (PI,P2, P........... P.} E S are normalized and treated as weights. Line 6 is our novel hybrid loss function discussed in Eq 10 to leverage the local ranking quality. The key step is to update the importance scores set based on current prediction errors and true ranking in Lines 7-8. We design score function shown in Line 8 to leverage the errors made in predictions and the priority true ranking for each location.
2 plnd-w31 v31
E, = T Σ
(12)
log2 +ry())
where r() denotes ranking function of the i-th location in the study area, and lv! - v the absolute difference between predicted injuries and true injuries. Basically, a higher score Ej indicates that

---

there are larger prediction errors and higher true ranking in this location s. Note that smaller r r(v)) denotes higher true ranking. To capture their geographical connections, we map the list of locations to their original locations on the grids, and then we apply a Gaussian filter to smooth the score distribution spatially Line 9. À is the standard deviation. In this way, the model can learn better spatially correlated patterns, and avoid over-focusing on a few standalone locations but ignoring the the area nearby, Next, the smoothed scores E" are normalized into P. where >. Pa = 1. Finally, we compute the gradient V f(0) by losShybrid and update model parameters 0. After few iterations, we obtain the learned model fa.
3.4 Complexity Analysis
In each iteration complexity of SpatialRank, we need to conduct forward propagation he(s; w), Vs E S U St and back propaaaopnn for computing Vhe(s; w), E S U S.. The complexity of forwarding is S = Eer ([S, [W(s) + [Stl)d & O(TSd), where d is the model size. To compute local DCG part, an extra cost on neighbor querying is needed replace location list size on its complexity. The cost of computing g (w;;x, S₁) is Ever |s, [|S S O(T5²) for NDCG and Ever [st (s) I (s) for local NDCG, The size of N2 is small, therefore the total cost is reduced to O(TSd + TS?).
4 Experiments
We perform comprehensive experiments three real-world traffic accident and crime datasets from Chicago and the State of lowa") Experiment results show that our proposed approach substantially outperforms state-of-art baselines on three datasets by up to 12.7%, 7.8%, and 4.2% in NDCG respectively. The case study and results on the state of lowa are shown in Appendix C.
Data. In the Chicago dataset, we collect data from the year 2019 to the year 2021. The first 18 months of this period are used as the training set, and the last 6 months of 2020 are used as the validating set. The year 2021 is used as testing set. The area of Chicago is partitioned by 500 m x 500 m square cells and converted to a grid with the size of 64 x 80. For the crime dataset, we use total crimes the risk score. For accident datasets, we use the number of injuries as the risk score. w Baselines. First, we use daily Historical Average (HA). Next, we consider popular machine-learning methods including Long Short-term Memory (LSTM) [16], and Convolutional LSTM Network (ConvLSTM) (31]. Thirdly, we compare with recent methods such as GSNet (37), Hetero- Peetero-ConLLS. ConvLSTM(40) and HintNet (5). Moreover, compared optimization approach with other NDCG optimization solutions including Cross Entropy (CE), ApproxNDCG (28), and SONG The details of the baselines are described in Appendix C.
29
Metrics. To measure the ranking quality of foremost locations, we test K E [30, 40, 50) on the test data. We use the metrics including NDCG, L-NDCG, and top-K precision (23) We report the average performance and standard deviation over runs for three datasets.
4.1 Performance Comparison
In table [ we can observe that our SpatialRank significantly outperforms other compared baselines both datasets. We observe a similar trend among all metrics. On both datasets, Hletro-CoovLLS LSTM and HintNet achieve similar results and outperform general machine learning methods such as LSTM and COMvLSTM. GSNet is designed on a dataset with a smaller grid size, thus performing worse on our larger grid. To study the efffcctiveness of learning the appropriate graph, we set the learning parameter / as a fixed ratio of 0.5 in SpatialRank # Oppositely, B is a parameter to be learned based temporal features in SpatialRank. In the Table and Table. 2 each * indicates that the performance improvement of the proposed method over this baseline is statistically significant based on the student t-test with a 0.05 over three runs. The results show that the design of the time-aware graph convolution is able to improve performance and capture more dynamic variations in the graph, therefore performing better over different top-k rankings.
2 https//ddata cityofchiccgo org/ Transportaiion Traitatiiiooiiiiiii Crashes Rrache/R5ca 13if
hutp://iicat gow!
7

---

TABLE PERFORMANCE COMPARISON
K=30
K=40
K=50
CHICAGO ACCIDENT
NDCG
PREC
L-NDCG
NDCG
PREC
L-NDCO
NDCG
PREC
L-NDCG
НА
.21400
.332.10
.50210
.25510
.322.0
.49.10
.235.00
.31610
.493:10
LSTM
CONVLSTM
225=
GSNET
TETERO- CONVLSTM
HINTNET
SPATIALRANK
.250-22
13%
5591
25
409.12..
553.1
2:1
.3.41.2 .585.1
SPATIALRANK
1%
1.% .621 268
.4001 .614
2% 278 1 .599
K=30
K=40
K=50
CHICAGO CRIME
NDCG
PREC
L-NDCG
NDCG
PREC
L-NDCG
NDCG
PREC
L-NDCG
.23:10 .348:00 .51410
.250+0 .33310 .506110 .259.0
.322.10
.449100
LSTM
CONVLSTM
GSNET
HETERO CONVLSTM
HINTNET
SPATIALRANK
.361+2u 18441%. 460- 05
SPAN
.373+2% ..9 46. .380 467 061.55x.6444555 34
B 0.5
%e: -10
TABLE 2: OPTIMIZATION COMPARISON
K=30
K=40
K=50
CHICAGO ACCIDENT
NDCG
PREC
L-NDCG
NDCG
PREC
L-NDCG
NDCG
PREC
L-NDCG
CE
.232.1 1% .88812%. 22%
.11..2xx .88812% .26611 1x. .400 1%.
.580.12%
APPROXNDCG
.240.12%
.597 .2%
15
.415.
.59116%
26441 1s.
.3981 1.
3%
SONG
.240=1%
2438 6610
254+2%
14
4400
.575
1%
SPATIALRANK
.257=1% .444 42.
268811
1.400 .614+2%
23%
403+1%
.599 1%
CHICAGO CRIME
K=30
K=40
K=50
NDCG
PREC
L-NDCG
NDCG
PREC
L-NDCG
NDCG
PREC
L-NDCG
CE
.362+1% .475 1s. 6534.%% 338 1% 666411%. .366+ .41+2% .644+2%
APPROXNDCG
.344=4% 3455 637 3444
421 336 .619
SONG
364=2% 3484 660+ .37:333799 466134444444433444544444444 45.+.% 39002902..4444 450 1649
SPANIALRANK
.373+2%
36
.644+6s. .64466
%e: x10-3
4.2
Optimization Comparison
To evaluate the effectiveness of our proposed hybrid objective function and importance-basee based location sampling, we perform experiments the same network architecture but with different optimization solutions including Cross Entropy (CE), ApproxNDCG, and SONG. The results are shown table Methods designed to optimize NDCG consistently perform better than Cross Entropy. SpatialRank substantially outperforms SONG and ApproxNDCG and made a noticeable improvement on L-NDCG as it considered in the objective function.
4.3 Ablation Study
We examine the effects of tuning hyper-parameter a in the hybrid loss function. We present results in table Recall that decides the ratio of L-NDCG takes in the objective function. means L-NDCG is not considered in the objective function. Starting from a = we observe trend that the overall performance improves stcadily while goes up, and it reaches the best performance

---

TABLE 3: ABLATION STUDY
K=30
K=40
K=50
CHICAGO ACCIDENT
NDCG
PREC
L-NDCG
NDCG
PREC
L-NDCO
NDCG
PREC
L-NDCG
0 0.0
0.251
0.439
0.610
0.263
0.407
0.612
0.274
0.289
0.605
0.05
0.239
0,416
0.559
0.254
0.394
0.600
0.266
0.386
0.597
0.1
0.256
0.443
0.621
0.268
0.421
0.608
0.276
0.400
0,599
0.2
0.250
0.444
0.616
0.261
0.411
0.605
0.271
0.394
0.603
0 0.3
0.234
0,422
0.606
0.243
0,388
0.59
0.253
0.374
0.591
CHICAGO CRIME
K=30
K=40
K=50
NDCG
PREC
L-NDKG
NDCG
PREC
L-NDCG
NDCG
PREC
L-NDCG
0 0.0
0.366
0.489
0.661
0.378
0.464
0.644
0.385
0.445
0.642
0 0.05
0.366
0.489
0.659
0.378
0.467
0.649
0.382
0.447
0.643
0.1
0.371
0.494
0.665
0.383
0.467
0.644
0.391
0.450
0.642
0.2
0.356
0.474
0.654
0.367
0.448
0.643
0.372
0.429
0.623
0.3
0.345
0.457
0.658
0.361
0.445
0.641
0.373
0.431
0.631
156
30
Fac Red
Herket
202
150
30
Figure 2: Comparison of cross-K function in Chicago Accident.
at equals 0.1 in the accident and crime dataset. It indicates that considering a reasonable weight of L-NDCG in the objective function boosts overall performance. These results demonstrate the importance of considering local ranking because NDCG, L-NDCG, and precision can be all improved.
4.4 Cross-K function
We use the Cross-K K function 14 with Monte Carlo Simulation 33 to evaluate the accuracy of predicted locations. The Cross-K function measures the spatial correlation between the predicted locations and true locations. Specifically, we calculate the average density of predictions within every distance d of true event in each day as shown Eq. 13
k(d)= x
Σ ifj
I(dy S dd/nn
(13)
where À is global density of event j. and I() is an identity function which equals one real distance de is smaller than d, else equals zero. is the number of events i. The results shown in Figure 2 The grey curve represents the complete spatial randomness and we use it as a reference baseline.
Higher is better. Our SpatialRank achieves the best predictions in both datasets, which indicates that the predictions of Raat are significantly spatially correlated with ground truth. The similar results the other two datasets shown in Appendix in the supplementary materials.
Additional Results and Case Study to demonstrate successful prediction examples are included in the Supplementary materials.

---

5 Related Work
Urban Event forecasting has been widely studied in the past few decades. Most early studies 6 [9] rely small-scale and single-source datasets with limited types of features, thus the prediction accuracy is limited. Notably, Zhou et al. [42 proposed a differential timevaryiig graph convolution network capturing traffic changes and improving prediction accuracy. Similarly, Wang et al. [37 proposed GSNet with a geographical module and weighted loss function to capture semantic spatial-temporal correlations among regions and solve data sparsity issues. Addressing the issue of spatial heterogeneity, Yuan et al. [40] proposed Hetero-Com ConvLSTM to leverage an ensemble of predictions from models learned from pre-selected sub-regions. Furthermore, An et al. (5] proposed HintNet partitions the study area hicrarchically and transfers learned knowledge over different regions to improve performance. However, most existing models rely on optimizing cross-entropy and the objective is to make accurate predictions on every location. Learning to rank an extensively studied area in recommendation systems and search engines [22] NDCG is widely adopted metric to measure ranking quality. The prominent class of methods involves approximating ranks in NDCG using smooth functions and subsequently optimizing the resultant surrogates. Taylor al. 34 tries to use rank distributions to smooth NDCG, but suffers from high computational cost. Similarly, Qin al. 28 approximates the indicator function by generalized sigmoid function with top-k variant. Noticeably, Qiu et al. [29] develop stochastic algorithms optimizing the surrogates for NDCG and its top-K K variant. Although is possible to directly apply the existing ranking methods the urban event ranking problem, the performance tends to be unsatisfactory as these methods commonly assume independence between items and queries and lack the ability handle spatiotemporal autocorrelation in the data.
13
21
6 Conclusion and Limitations
In this work, we formulate event forecasting as a location ranking problem. We propose a novel method SpatialRank to learn from spatiotemporal data by optimizing NDCG surrogates. To capture dynamic spatial correlations, we design an adaptive graph convolution layer to learn the graph from features. Furthermore, regions, and a novel ranking-based importance sampling mechanism to leverage the importance of each location considered during the model training. Extensive experimental results on three real-world datasets demonstrate the superiority of SpatialRank compared to baseline methods.
propose hybrid loss function capture potential risks around hot-spot
Limitations: the model's performance might be affected by other properties of data, such as spatial heterogeneity and sparsity. We observe less improvement over baselines on the Iowa dataset, partially due to that the data is sparser over large area with heterogeneity. These are issues addressed by some of the prior work and can be addressed in our future work. In addition, the new algorithm increases the training time complexity due to the sampling steps. This is acceptable due to the relatively small number of locations and time periods in urban event datasets but may require extra work to generalize to large datasets.
Acknowledgments and Disclosure of Funding
T. Yang was partially supported by NSF Career Award 18844403, NSF Fair AI Grant 2147253, NSF RI Grant 21 10545.
References
[1] Chicago Police Department, htwp:r/ttaattttielllooeeeeeenees 2020. show. 2022.
[2] Fox News,
[3] Police Executive Research Forum, http:////wwwwwgtttoooooooo//ebee 2022.
[4] J. Abellán, G. López, and J. De Oña. Analysis of traffic accident severity using decision rules via decision trees. Expert Systems with Applications, (1111)5)56077655545 2013.
10

---

[5] B. An, A. Vahedian, X. Zhou, W. Street, and Y. Li. Hintnet: Hierarchical knowledge transfer networks for traffic accident forecasting on heterogeneous spatio-temporal data. Proceedings the 2022 SIAM International Conference on Data Mining, 2022.
[6] I.. Barba, N. Rodríguez, and C. Montt. Smoothing strategies combined with arima and neural networks improve the forecasting of traffic accidents. The Scientific World Journal, 2014, 2014.
R. Bergel-Hayat, M. Debbarh, C. Antoniou, and G. Yannis. Explaining the road accident risk: Weather effects. Accident Analysis & Prevention, 60:456-455 465, 2013.
[8] Bhatia, H. Jain, Kar, Varma, and P. Jain. Sparse local embeddings for extreme multi-label classification. In Advances in Neural Information Processing Systems, volume 29, pages 730-738, 2015.
[9] C. Caliendo, M. Guida, and A. Parisi. A crash-prediction model for multilane roads. Accident Analysis Prevention, (4)6657766,, 2007.
[10] R. N. Carey and K. M. Sarma. Impact daylight saving time on road traffic collision risk: a systematic review. BMJ Open, 7(6), 2017.
[11] L.-Y. Chang. Analysis of freeway accident frequencies: negative binomial regression versus artificial neural network. Safety science, 43(8))541-555) 2005.
[12] L.-Y. Chang and W.-C. Chen. Data mining of tree-based models to analyze freeway accident frequency. Journal of safety research, 36(4):365-375, 2005.
[13] M. M. Chong, A. Abraham, and M. Paprzycki. Traffic accident analysis using decision trees and neural networks. ArXiv, cs.4400050000, 2004.
[14] P. M. Dixon. Ripley's K Function, John Wiley & Sons, Ltd, 2014.
[15] N. Dong, H. Huang, and I.. Zheng. Support vector machine in crash prediction at the level of traffic analysis zones: Assessing the spatial proximity effects. Accident Analysis & Prevention.
[16] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735- 1780, 1997.
[17] K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems. 20(4):422 446, 2002.
[18] A. Khezerlou, X. Zhou, L. Li, Z. Shafiq, A. Liu, and Zhang. A traffic flow approach to early detection of gathering events: Comprehensive results. ACM transactions intelligent systems and technology. 8(6):1 24, 2017.
[19] A. V. Khezerlou, X. Zhou, X. Li, W. N. Street, and Y. Li. Dilsat: Predicting urban dispersal events through deep survival analysis with enhanced urban features. ACM transactions intelligent systems and technology, 12(4):1225, 2021.
[20] Z. Li. C. Huang, Xia, Xu, J. Pei. temporall hypergraph self supervised learning crime prediction. arXiv.rrg Ithaca, 2022. Cornell University Library, AiX.voorg.
[21] L. Lin, Q. Wang, and A. Sadek. A novel variable selection method based on frequent pattern tree for real-time traffic accident risk prediction. Transportation Research Part C, 55, 2015.
[22] T.-Y. Liu. Learning to Rank for Information Retrieval, Springer, 2011.
[23] Lu, C. Xu, W. Zhang, L. Duan, and T. Mci. Sampling wisely: Deep image embedding by top-k precision optimization. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 7960-7966 IEEE, 2019.
[24] X. Mao, C. Yuan, J. Gan, and S. Zhang. Risk factors affecting traffic accidents at urban weaving sections: Evidence from china. International journal of environmental research and public health, 16(9):1542, 2019.
[25] T. R. Miller, M. A. Cohen, D. I. Swedler, B. Ali, and D. V. Hendrie, Incidence and costs of personal and property crimes in the usa, 2017. Journal of benefit-com cost analysis, 12(1):24 54, 2021.
[26] A. Najjar, S. Kaneko, and Y. Miyanaga. Combining satellite imagery and open data to map road safcty, In Thirty-First AAAI Conference Artificial Intelligence, 2017.
[27] NHTSA. 2021 fatality data show increased traffic fatalities during pandemic. https://www wwtsan / traffic-fatalltieee
press-releases// heaee/aarlll-aalla eettyy--tttttt--eeeeee 2021- traffitraf----- fatalities Jun 2021.
11

---

[28] T. Qin, T.-Y. Liu, and H. Li. A general approximation framework for direct optimization of information retrieval measures. Information Retrieval, 13(4)3375-377, 2010.
[29] Z. Qiu, Q. Hu, Y. Zhong, I. Zhang, and T. Yang. Large-scale stochastic optimization of ndeg surrogates for deep learning with provable convergence. In International Conference Machine Learning, 2022.
[30] H. Ren, Y. Song, J. Wang. Y. Hu, and J. Lei. A deep learning approach to the citywide traffic accident risk prediction. 2018 21st International Conference Intelligent Transportation Systems (ITSC), pages 3346 3351, 2018.
[31] X. Shi, Z. Chen, H. Wang, D.-Y. Yeung, W.-k. Wong, and W.-c Woo. Convolutional Istm network: machine learning approach for precipitation noweasting. 2015.
[32] R. Swezey, Grover, B. Charron, and S. Ermon. Pirank: Scalable learning to rank via differentiable sorting, Advances in Neural Information Processing Systems, 34, 2021.
[33] R. Tao and J.-C. Thill. Flow cross k-function: bivariate flow analytical method. International journal of geographical information science IJGIS, 33(10):0055-071, 2019.
[34] M. Taylor, J. Guiver, S. Robertson, and T. Minka. Softrank: optimizing non-smooth rank metrics. In Proceedings of the 2008 International Conference on Web Search and Web Data Mining, pages 77-86. 2008.
[35] R. Tobler. A computer movie simulating urban growth in the detroit region. Economic geography, 46:234-240, 1970.
[36] A. Vahedian, X. Zhou, Tong, W. Street, and Y. Li. Predicting urban dispersal events: A two-stage framework through deep survival analysis on mobility data. 2019.
[37] B. Wang, Y. Lin, Guo, and H. Wan. Gsnet: Learning spatial-temporal correlations from geographical and semantic aspects for traffic accident risk forecasting. In 2021 AAAI Conference on Artificial Intelligence (AAAI'21), 2021.
[38] M. Wu, Y. Chang, Z. Zheng, and H. Zha. Smoothing deg for learning to rank: a novel approach using smoothed hinge functions. In Proceedings of the 18th ACM conference information and knowledge management, CIKM '09, pages 1923 1926. ACM, 2009.
[39] Z. Wu, S. Pan, G. Long, J. Jiang, X. Chang, and C. Zhang. Connecting the dots: Multivariate time series forecasting with graph neural networks. In Proceedings the 26th ACM SIGKDD international conference on knowledge discovery data mining, pages 753-763, 2020.
[40] Z. Yuan, X. Zhou, and T. Yang. Hetero-convistm: deep learinng approach traffic accident prediction on heterogeneous spatio-temporal data. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 984-992, 2018
[41] Y. Zhang, Y. Li, Zhou, Kong, and Luo. Trafficgan: Off-deployment traffic estimation with traffic generative adversarial networks. 2019 IEEE International Conference on Data Mining (ICDM), pages 2019.
[42] Z. Zhou, Y. Wang. X. Xie, L. Chen, and C. Zhu. Foresee urban sparse traffic accidents: A spatiotemporal multi granularity perspective. IEEE Transactions on Knowledge and Data Engineering, pages 1-1, 2020.
12


Reviews:
[{"review_id": "g6HH4Z7mau", "reviewer": "Reviewer_RZQE", "summary": "This paper investigates the problem of future urban event prediction on spatiotemporal data. This is an important problem for a broad range of urban application. Different from prior work, this paper for the first time predicts most likely future events by directly optimizing location ranking in the prediction through NDCG optimization. The authors propose a dynamic adjacency matrix of locations, a hybrid NDCG loss function and a spatial sampling algorithm to handle the unique challenges brought by spatiotemporal data. Experimental results on three datasets show that the proposed solution can beat state-of-the-art event prediction models as well as existing NDCG optimization solutions on crime and accident prediction. ", "questions": "1.\tWhat is the model architecture used for the optimization comparison experiment? How does the model architecture choice affect the performance of the optimizers? Is it possible that with a different network architecture the other optimizers such as SONG or approxNDCG can beat spatialRank?\n2.\tHow is top-k precision defined in the experiments?\n", "limitations": "N/A.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe problem investigated by the paper is significant for many urban applications such as crime prediction and accident forecasting. \n2.\tThe paper is the first to use NDCG optimization on spatiotemporal data for location ranking. This is a novel idea for event prediction.\n3.\tThe work also adds value to the literature of ranking algorithm for providing solutions on how to handle non-iid spatial data ranking. \n4.\tThe experiments show that the proposed SpatialRank model outperforms not only event/accident prediction methods but also traditional NDCG optimization method. This suggests that the model is effective in addressing some of the unique challenges in spatiotemporal data.\n", "weaknesses": "1.\tAs discussed by the authors, the hybrid loss function might introduce significant computation cost increase to the learning algorithm. In addition to the complexity analysis, the authors should provide additional evidence (e.g., experiments) to justify the impact to training time.\n\n2.\tThere are quite a lot of symbols in the paper and the authors should provide a table or summary of these symbols. Without such information, the complexity analysis part is a bit hard to follow.\n\n3.\tThe experimental results should be presented with error bars. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer RZQE,\n\nThank you very much for your comments and appreciation of our paper! Below we address your questions and concerns.\n\nQ: In addition to the complexity analysis, the authors should provide additional evidence (e.g., experiments) to justify the impact to training time.\n\nA: Thank you for the suggestion! We have added comparisons with SOTA methods on average training time in seconds per epoch. The results are copied below and will be added to the revision. The Chicago crime dataset and the Chicago accident dataset have the same input feature; thus, have equivalent training costs. In summary, SpatialRank trains faster than two SOTA baselines HintNet and GSNet on both datasets. It is only slower than HeteroConvLSTM but the training times of the two are on the same order of magnitude. Given the improvement in prediction performance, we believe this is an acceptable cost, which will not affect the practical value of the proposed method.\n| Dataset      | SpatialRank | HintNet | HeteroConvLSTM | GSNet |\n| ----------- | ----------- |----------- | ----------- | ----------- |\n| Chicago     | 88.2      | 132.1 | 47.7 | 98.8 |\n| Iowa   | 76.5      | 117.5 | 41.6 | 83.5 |\n\n* * *\n\nQ: There are quite a lot of symbols in the paper and the authors should provide a table or summary of these symbols.\n\nA: Thank you for the suggestion. We will add a symbol table to the revision. Besides, explanations on symbols related to ranking optimization are included in Section 3.2 and Section 4.2. Symbols for problem formulation are introduced in Section 3.1.\n| Symbol | Explanations | \n| ----------- | ----------- |\n| $S$     | Spatial filed, study area | \n| $s$     | A partitioned location, grid cell| \n| $T$     | Temporal filed, study period| \n| $t$     | Time interval (e.g. hours, days)| \n| $F_T$     | Temporal features (e.g. weather, time)| \n| $F_S$     | Spatial features (e.g. POI) | \n| $F_{ST}$     | Spatiotemporal features (e.g. traffic conditions) | \n| $y$     | Event Risk Score| \n| $Z$     | Discounted Cumulative Gain (DCG) score| \n| $E$     | Node embedings| \n| $a$     | Pearson correlation coefficient| \n| $NDCG$     | Normalized Discounted Cumulative Gain| \n| $L-NDCG$     | Local Normalized Discounted Cumulative Gain| \n| $Prec$     | top-K precision| \n| $r()$     | Ranking function| \n| $N()$     | Neighbour querying| \n\n\n\n* * *\n\nQ: The experimental results should be presented with error bars\n\nA:  Thank you for pointing it out. We have revised the three experiment results and now report the average performance including NDCG, L-NDCG, and Precision with standard deviation over 3 runs in Table 1 and Table 2 of the updated pdf. The results show that SpatialRank significantly outperforms other compared baselines in three datasets. SpatialRank substantially outperforms SONG and ApproxNDCG and made a noticeable improvement on L-NDCG as it is considered in the objective function.\n\n* * *\n\nQ: What is the model architecture used for the optimization comparison experiment? How does the model architecture choice affect the performance of the optimizers? Is it possible that with a different network architecture the other optimizers such as SONG or approxNDCG can beat spatialRank?\n\nA:  The model architecture used for optimization comparison is described in Section 4.1, which utilizes LSTM to capture temporal dependencies and graph convolution layers to capture spatial dependencies. This is consistent with the model proposed in Section 3.1. . In the ablation study, we demonstrate that with the same network architecture but different loss functions and learning algorithms, SpatialRank can still beat other baselines (e.g., CE SONG). This proves that the improvements in performance came not solely from the network architecture proposed in 3.1 but all the three contributions. In fact, our proposed framework can improve the performance of other networks architectures as well, which is our core contribution. \n\n* * *\n\nQ: How is top-k precision defined in the experiments?\n\nA:  Top-k precision is a widely accepted measurement in learning-to-rank problems. In our cases, it is equal to the number of locations in top-k recommendations where events occurred divided by the number of recommendations k. It indicates how well our model can capture events within our recommendations. More explanations can be found in this paper Lu et al[1]. We will also cite this paper in the experiment section in the revision.\n\n* * *\n\n[1] Lu, Jing, et al. Sampling Wisely: Deep Image Embedding by Top-K Precision Optimization. IEEE, 2019\n\n\n"}, {"review_id": "foxfhV5j1Z", "reviewer": "Reviewer_yV4P", "summary": "The paper proposes a deep learning model called SpatialRank that predicts the top-k riskiest locations of future events such as traffic accidents and crimes by optimizing a spatial version of the NDCG measure. The model features adaptive graph convolution layers that learn the spatiotemporal dependencies from data, a hybrid loss function that balances global and local ranking quality, and an importance-sampling with spatial filtering algorithm that guides the model to focus on important locations. The model is evaluated on three real-world datasets from Chicago and Iowa, and outperforms several methods in terms of NDCG, L-NDCG, and precision. The model also demonstrates better spatial correlation with ground truth using the cross-K function.", "questions": "Q1. Directly constructing from temporal events is more effective and explicit. Why generating a dynamic graph from features is necessary or effective? How does it capture the dynamic spatiotemporal dependencies among locations? How does it compare with directly constructing a graph from temporal events?\n\nQ2. Why the method set E1 = E2 = FST in Eq. (4) and Eq. (5)? Since these two equations are almost same, why combine the two embedding Z_1 and Z_2 in Eq. (6) for generating the dynamic adjacency matrix?", "limitations": "The authors have discussed the limitations of their work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "S1 The proposed method is evaluated on three different real-life datasets.\n\nS2 The idea of introducing the NDCG metric into a location-based learning problem is somewhat interesting.", "weaknesses": "Though this paper arises an interesting problem, I have the following major concerns about this paper:\n\n(1) The motivation for urban event ranking is weak. The paper does not highlight the significance and novelty of this problem, and why it is more important than making event predictions for each location. The paper should provide more evidence or examples to motivate the need and value of ranking locations for future events.\n\n (2) The technical contribution is limited and the model is not novel enough. The paper does not clearly state how the proposed model differs from existing methods in terms of problem formulation, model design, optimization strategy, or evaluation metric. The designed NDCG loss function is quite similar to Eq. (3). I do not think there is much difference between these two functions. The paper should provide more details and analysis to demonstrate the advantages and challenges of the proposed approach.\n\n(3) The model uses a Euclidean distance to define the neighborhood of each location, which may not reflect the actual spatial proximity or connectivity of locations in terms of road network or travel distance.\n\n(4) The presentation and description of the framework is poor and incomplete. There are several unclear or confusing points in the paper, please refer to the above Questions.\n\n(5) The experiments are not sufficient. There are several limitations or missing details in the experimental section, such as:\n(5.1) No significance test to show whether the differences between SpatialRank and baselines are statistically significant or not.\n(5.2) No ablation study experiments to show the effectiveness or necessity of each component of SpatialRank, such as the dynamic graph generation module or the designed NDCG loss function.\n\n- Minor concerns: \n\n(6) There are some grammar issues in the paper, such as “learning-to-ranking” in Line 51, which should be corrected.\n\n(7) The paper lacks a framework figure to illustrate the overall architecture and workflow of the model, which makes it hard to follow and understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer yV4P\n\nThank you very much for your comments! Please find our responses below:\n\nQ: The motivation for urban event ranking is weak. The paper should provide more evidence or examples to motivate the need and value of ranking locations for future events.\n\nA: We believe being able to correctly rank the foremost important locations meets the real-world demand because many related works and reports have shown that deploying limited staffing resources to the most needed places is a crucial task for law enforcements. Case studies also show that our method can find the riskiest locations and capture more hotspots than baselines who predict for all locations. **We provide more details in response #2 in the overall rebuttal.**\n* * *\nQ:  The technical contribution is limited and the model is not novel enough. The paper should provide more details and analysis to demonstrate the advantages and challenges of the proposed approach.\n\nA: We believe this paper presents important, novel, and valid contributions. \n\nFirst, this is the first paper to formulate an urban event forecasting problem as a spatial learning-to-rank problem and solve it by directly optimizing a spatial version of the NDCG measure.\n\nSecond, we propose a novel local ranking measurement named L-NDCG and integrate it into our new loss function. This is to the best of our knowledge the first NDCG-based measure that considers spatial autocorrelation of data. The new loss **is different from the original NDCG** in not only an additional local ranking quality term, but also the underlying scientific assumptions and the non-trivial computational techniques needed to efficiently evaluate it. \n\nThird, we propose a novel importance-based location sampling algorithm to efficiently train the model to optimize the hybrid NDCG loss function. **The algorithm is also very different from traditional NDCG optimization techniques** as it uses spatial sampling to address the L-NDCG part of the loss function for the first time.\n\nIn addition, we propose a network architecture with a novel adaptive convolution layer to capture the dynamic correlations between locations. \n\nFinally, we provide comprehensive experiments and evaluate the improvement gained through each of the above contributions. \n\n**More details in response #3 of overall rebuttal.**\n\n* * *\n\nQ:   The model uses a Euclidean distance to define the neighborhood of each location, which may not reflect the actual spatial proximity or connectivity of locations in terms of road network or travel distance.\n\nA:  We choose Euclidean distance because: \n* Euclidean distance is a widely accepted distance measure in the field of spatiotemporal forecasting problems, because it is computationally efficient and generally effective [1].\n* Computing actual spatial proximity in terms of if road network is computationally expensive and requires additional data, which might not be available to users. \n* The difference between using Euclidean distance and using travel distance is negligible in the small local neighborhood we consider. We computed the travel distance between the centroids of grid cells using road information, and found that the Euclidean distance is proportionally equivalent to the travel distance under current partition granularity in our Chicago and Iowa datasets. Even using Euclidean distance our model is already superior to SOTA methods.\n* * *\n\nQ:   There are some grammar issues in the paper.\n\nA:  Thank you for pointing this out. We will correct grammar issues.\n\n* * *\n\nQ:   The paper lacks a framework figure to illustrate the overall architecture and workflow of the model, which makes it hard to follow and understand.\n\nA:  Due to the limited paper space, the framework figure and extra experiment results were in supplementary material. We will improve the presentation in the revision.\n\n* * *\n\nQ:    Why the method set E1 = E2 = FST in Eq. (4) and Eq. (5)? Why combine the two embedding Z_1 and Z_2 in Eq. (6) for generating the dynamic adjacency matrix?\n\nA:   Sorry for the confusion. If nodes’ information is not available, $E_1$ and $E_2$ are randomly initialized node embeddings to be learned during training. In our case, we take advantage of two sets of $F_{ST}$ from the source node and target node respectively to represent node embeddings $E_1$ and $E_2$, so we are calculating the pairwise similarity between source node features and target node features in Eq. 6. The subtraction and ReLU activation function in Eq. (6) lead to the asymmetric property. In other words, we treat nodes' spatiotemporal features as embeddings to reveal underlying dynamic connections between nodes. We will rephrase these equations in the revision.\n* * *\n\nQ:    Directly constructing from temporal events is more effective and explicit. Why generating a dynamic graph from features is necessary or effective? How does it capture the dynamic spatiotemporal dependencies among locations? How does it compare with directly constructing a graph from temporal events?\n\nA:   Our proposed graph learning method learns graphs based on nodes’ features and dynamically updates graphs based on temporal information. As mentioned in Section 3, we use historical events to generate a static graph and learn a time-variant graph from $F_{ST}$ (e.g., traffic volume) through different periods. This design can capture the dynamic dependencies among locations in the real-world datasets. The experiment results in Table 1 further proves the benefits of considering dynamic graph versus static graphs. We are not very sure what you mean by constructing a graph from temporal events. The dataset covers a long time period (e.g., a few years). With events from each location during each day as nodes (~ millions), the graph, in particular the adjacency matrix could be too huge for any training algorithm.  \n\n* * *\n[1] Zhe Jiang. 2018. A survey on spatial prediction methods. TKDE 31, 9 (2018), 1645–1664\n"}, {"review_id": "La28ulf0t6", "reviewer": "Reviewer_JmsG", "summary": "The paper proposes a method which applies learning to rank losses to spatiotemporal event prediction. The observed data is considered over a discrete spatial partitioning and considers a time series of feature information. Features are grouped into purely temporal, purely spatial (not-time dependent) and spatiotemporal information. In addition, there is a risk score >0 for time, location pairs where an event occurred. The examined task is now to predict the top-k riskiest location for the next time step.\nThe method encodes the state of the network using a know spatiotemporal prediction network which employs graph NN layers to encode the underlying spatial connectedness modelling a road network. \nThe paper's novelty is to learn a dynamic adjacency matrix from the spatiotemporal feature of each location, which allows us to dynamically model the current conditions, such as traffic.  This dynamic adjacency matrix is combined with the static one.\nA second novelty is adapting the ranking score to consider local rankings, which replace the global rank of location the original NDCG with a ranking score within a local neighbourhood. The localized NDCG is also combined with global NDCG loss.\nFot training, the authors propose to compute a weight for each element which is based on the ranking score of the  prediction error.\n", "questions": "What is the intuition of subnetwork for the dynamic adjacency matrix, and what is it supposed to model? Can you give more details about the intuition of the architecture and how it naturally aligns with the static adjacency matrix? \n(There should not be a link if there is no connection in the static matrix, right?).\n\nCan you give details on how much the instance weighting in training helped to improve the performance or speed up the conference? \n\nSimilarly, is there an ablation study on the dynamic adjacency matrix?\n\n", "limitations": "The paper addresses its limitations in a dedicated paragraph.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper proposes a novel idea to compute a ranking of locations instead of directly computing an event probability or even the next event, which might be hard to measure. In other words, instead of predicting the event likelihood, a ranking is learnt, which should be easier to learn but sufficient for various tasks.\n\nThough the general layout of the proposed method is to combine NDCG with a spatial prediction backbone, the paper adds three modifications and shows that they considerably improve the performance compared to baselines.\n\nThe paper is good to follow and provides a reproducible description of the proposed methods.\n\nThe experiments show improved results von two real-world data sets, and the authors provide an ablation study.\n\nThe paper provided sufficient supplementary material to check all the details.", "weaknesses": "Though I could follow the reasoning of computing a top-k query, the motivation why this information is enough in several applications could be motivated better in the introduction.\n\nThe ablation study seems only to compare various values of \\delta. But it would also be interesting to see the impact of the other contributions. As the technical contribution seems to consist of three relatively independent adaptions to the spatiotemporal setting seeing how much each contribution added to the improvement might be very insightful. \n\nThe description of the dynamic adjacency matrix is a little bit cryptic. In particular, it is unclear how it is trained and why it should represent a dynamic adaption of the static adjacency matrix", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer JmsG,\n\nThank you very much for your comments and appreciation of our paper! Below we address your questions and concerns.\n\nQ: Though I could follow the reasoning of computing a top-k query, the motivation why this information is enough in several applications could be motivated better in the introduction.\n\nA: Thanks for your suggestions! We will add extra explanations and citations on our motivation for formulating a ranking problem in the introduction. We believe being able to correctly rank the foremost important locations meets the real-world demand. Given limited law enforcement resources such as staffing, the most essential task is to ensure they are allocated to the riskiest areas. The value k can be adjusted by users and set larger to report more locations. Our case study in Figure 4 demonstrates that our proposed method can capture more hotspots than baselines who make predictions for all the locations. \n**Please also see more details in response #2 in overall rebuttal.**\n\n***\nQ: The ablation study seems only to compare various values of \\delta. But it would also be interesting to see the impact of the other contributions.\n\nA: Sorry for the confusion. Due to the page limit and massive experiment results, instead of having independent tables, we concatenate some of ablation studies into performance comparison and optimization comparison part. To study how adaptive convolution layer improves the performance, we set $\\beta = 0.5$ in SpatialRank marked as SpatialRank* in performance comparison part in Table 1, so that ratio $\\beta$ between static adjacency matrix and dynamic matrix is fixed. To study how our proposed optimization solution contributes to the performance, we compare our approach with other SOTA solutions on the same network architecture in the optimization comparison part in Table 2. All the results show that each of the three contributions of the paper plays an important role in the superior performance of SpatialRank.\n\n***\n\nQ: It is unclear how it is trained and why it should represent a dynamic adaption of the static adjacency matrix. What is the intuition of subnetwork for the dynamic adjacency matrix, and what is it supposed to model? Can you give more details about the intuition of the architecture and how it naturally aligns with the static adjacency matrix? (There should not be a link if there is no connection in the static matrix, right?).\n\nA:  Dynamic adjacency matrix is learned from computing pairwise similarity between source node embeddings and target node embeddings. If nodes’ information is not available, $E_1$ and $E_2$ are randomly initialized node embeddings to be learned during training, we treat nodes’ $F_{ST}$ as embeddings in our case. Parameter $W_1$ and $W_2$ are learnable parameters. Intuitively, the static adjacency matrix indicates a baseline correlation between different locations. For example, event patterns in a residential area might be constantly correlated with patterns in a shopping center nearby. Static adjacency matrix is pre-computed before training. This is also what most of the related work has been done. However, inspired by many observations from related studies [1], it is evident that we think this correlation is not always constant. Therefore, we use locations’ time-variant features to construct a new adjacency matrix, and this dynamic adjacency matrix varies with time. We learn the parameters in this dynamic matrix during the network training process. Finally, we combine the static and the learned dynamic matrices through a learned weight (\\beta). In this way, a combined adjacency matrix can be treated as adaptation from a static adjacency matrix considering the influence of other features during different periods. There should not be a link between two nodes in static matrix if the Pearson correlation coefficient between them is zero. The link between locations of the final adjacency matrix is determined by both the static and the dynamic graphs.\n\n***\n\nQ:  Similarly, is there an ablation study on the dynamic adjacency matrix?\n\nA:  As we answered in the second question, there are ablation studies on the learned combined adjacency matrix versus fixed adjacency matrix in table 1. Results show that introducing the dynamic adjacency matrix can effectively improve the performance of the model (last two rows in Table 1). There could be other ways to construct the dynamic adjacency matrix, but we did not perform additional evaluations on these choices as this is not our main focus of the paper.\n***\n\nQ:  Can you give details on how much the instance weighting in training helped to improve the performance or speed up the conference?\n\nA:  Thanks for pointing this out. We will add an extra ablation study to the camera-ready paper and copied below. We compare SpatialRank to the same model with equal weights. The results show that instance weighting constantly improves the prediction performance.\n\n| Dataset | methods| NDCG@30 | PREC@30 | L-NDCG@30 | NDCG@40 | PREC@40 | L-NDCG@40 | NDCG@50 | PREC@50 | L-NDCG@50 |\n| ----------- | ----------- | ----------- |----------- | ----------- | ----------- |----------- | ----------- |----------- |----------- | ----------- |\n| Chicago Accident | equal-weights  | .255      | .441 | .622 | .265      | .417 | .613 |.274      | .401 | .595 |\n| Chicago Accident    |  SpatialRank  | .257      | .444 | .621 | .268      | .420 | .614 |.278      | .403 | .599 |\n| Chicago Crime  |  equal -weights    | .373      | .480 | .660 | .379      | .466 | .456 |.390      | .450 | .649 |\n| Chicago Crime |  SpatialRank  | .373      | .491 | .665 | .380      | .467 | .647 |.392      | .446 | .644 |\n| Iowa  |  equal -weights  | .531      | .304 | .617 | .557      | .264 | .591 |.573      | .225 | .546 |\n| Iowa  |  SpatialRank  | .540      | .309 | .618 | .563      | .268 | .600 |.585      | .232 | .550 |\n\n***\n\n\n[1] Carey et al “Impact of Daylight Saving Time on Road Traffic Collision Risk: a Systematic Review.”\n"}, {"review_id": "gD2DgYxnNS", "reviewer": "Reviewer_VBY2", "summary": "This paper casts event forecasting as location ranking problem. They propose a spatial event ranking approach called SpatialRank. SpatialRank optimizes the NDCG metric while taking spatiotemporal autocorrelation into account. \n\nSpatialrank uses a graph convolutional network to encode autocorrelated input and applies NDCG optimization on the top fully-connected layer. Based on the experiments on 3 datasets, it outperforms the underlying approaches on precision@k metric.", "questions": "* What are the quantitative measures of performance wrt baselines?\n* Do subregions need to be given?", "limitations": "Authors have addressed a few limitations including performance, data sparsity. To stress-test the approach the authors are advised to test their method on more variety of datasets. One can't confidently enumerate all limitations before it is tested on many datasets.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper is a good application of graph convolutional networks to autocorrelated spatiotemporal data to solve the event ranking problem. None of the individual pieces are unique, but the compound technique is a novel application.\n\n* Introduction of L-NDCG is novel as it addresses the problem of hotspots dominating the overall NDCG metric.\n* It is also nice that the authors performed an ablation study -- albeit being rather minimal.\n* Ideas are fairly clear and explained well.\n* The work might be significant if its applicability, which is teased in Section 6, is discussed in more detail. ", "weaknesses": "A major weakness of the paper is its presentation. It needs attention to detail as well as how it is presented overall before being ready to be published. Some of the presentation issues are as follows:\n* Lines 41-61 is basically a Related Work section. However, the reader is greeted with an actual Related Work section on the same page where most of the information is repeated.\n* Line 151. What's Adam-style optimizer? Do you mean to say gradient-based optimizers?\n* The main contribution of the paper is illustrated in Figure 1 of the Appendix. It deserves to be in the actual paper.\n* Iowa dataset needs to be in the paper and the existing tables should be squeezed as they don't carry too much information to deserve that much space.\n* The paper occasionally leads the reader to assume that the actual task is to increase the NDCG score although the problem is to predict top k events.\n* A few spelling, capitalization, or other errors:\n  * Line 86: Related Works -> Related Work\n  * Line 182: Relu -> ReLU\n  * Algorithm 1: Normorlize -> normalize\n  * Line 280: We -> we\n  * Line 281: Convlstm -> ConvLSTM\n  * K@30 -- can't really tell this means. It is maybe K=30?\n\nOne other main weakness of the paper is its over-reliance on NDCG. First of all, experiments should't compare the approaches based on NDCG as it is the metric this approach is optimizing for as it wouldn't make a fair comparison per Goodhart’s law. Secondly, it is stated that \"NDCG is not a perfect metric for our problems as it neither measures the local ranking quality nor considers spatial autocorrelation among locations.\" However, NDCG is still treated as somewhat ground truth throughout the paper.\n\nOnce the problem is converted into NDCG optimization, other approaches can also be utilized. For the experiments, tree-based ranking models can also be evaluated to replace the final FC layer as they have the potential to perform better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer VBY2,\n\nThank you very much for your comments! Below we answer your questions and address your comments.\n\nQ: The reader is greeted with an actual Related Work section on the same page where most of the information is repeated.\n\nA: Thank you for your suggestions! We will rephrase the related work section and reduce the repeated information in revision.\n\n***\nQ: What's Adam-style optimizer? Do you mean to say gradient-based optimizers?\n\nA: Sorry for the confusion. We mean the Adam optimizer [1], which is an extension to stochastic gradient descent optimizer. Adam optimizer computes individual adaptive learning rates for different parameters from estimates of the first and second moments of the gradients. We found that Adam optimizer performs better in our datasets, so we use this term in the paper. Theoretically, equation 3 can be optimized by any gradient-based optimizer. To be accurate, we will use ‘gradient-based optimizers’ in our updated paper.\n\n***\n\nQ: Figure 1 deserves to be in the actual paper.\n\nA: Thank you for the suggestion. Figure 1 will be moved to Section 3 in the revision.\n\n***\n\nQ: Iowa dataset needs to be in the paper and the existing tables should be squeezed.\n\nA: Thank you for the suggestion. Given this cross-domain spatiotemporal ranking problem, we feel the need to use multiple baselines, metrics, and analysis methods from both learning-to-rank and spatiotemporal event forecasting domains to guarantee the completeness and consistency of the experiments. We agree with you that including results from a different geographic area will make the conclusion of the paper stronger. We will rearrange the materials to fit the results of the Iowa data. **Please check the attached pdf in the overall rebuttal**\n***\nQ: The paper occasionally leads the reader to assume that the actual task is to increase the NDCG score although the problem is to predict top k events.\n\nA: To clarify, the goal is to predict the top k events with the highest risk scores. We use a hybrid NDCG score as the objective function to evaluate the ranking quality, where optimizing (maximizing) the hybrid NDCG will generally lead to a model that can better rank the top events. The traditional NDCG score is part of the objective function but simply maximizing itself might not give the best ranking results due to spatial autocorrelation of the data. This is the motivation for us to propose the hybrid NDCG loss with the local NDCG term and the new algorithm.\n***\nQ: A few spelling, capitalization, or other errors.\n\nA: Thank you for pointing them out! We will correct all issues in the revision.\n\n***\n\nQ: K@30? maybe K=30?\n\nA: Yes, it is correct. For example, NDCG K@30 means only top-30 ranked locations contribute to the final NDCG score. Details can be found in Experiments Section.\n\n***\n\nQ: One other main weakness of the paper is its over-reliance on NDCG. experiments should't compare the approaches based on NDCG as it is the metric this approach is optimizing for as it wouldn't make a fair comparison per Goodhart’s law.\n\nA: Firstly, NDCG is only one of the three metrics used in our experiments. We also compared top-k precision and local NDCG. NDCG and top-k precision are the most widely accepted metrics in learning to rank problems. Secondly, our method does not directly optimize any of the individual metrics. It indeed optimizes a hybrid NDCG, which is the combination of NDCG and L-NDCG. By contrast, some of the SOTA baselines (such as SONG and ApproxNDCG) directly optimize NDCG as their objective. Therefore, the comparison is actually unfair to our model. However, SpatialRank still achieves superior performance in all the three metrics, including NDCG, compared to these methods. Lastly, the motivation of ranking locations based on importance is to meet the real-world demand that deploying limited law enforcement resources to the most needed places. Existing methods fail to leverage the foremost important locations and thus perform poorly in terms of ranking quality.\n***\n\nQ: It is stated that \"NDCG is not a perfect metric for our problems as it neither measures the local ranking quality nor considers spatial autocorrelation among locations.\" However, NDCG is still treated as somewhat ground truth throughout the pape?\n\nA: We believe NDCG is a reasonable measurement in this problem. Though we believe the proposed L-NDCG is more suitable for this problem, it is more convincing to also include existing widely accepted ranking measures such as NDCG and top-k precision. We are not using it as a ground truth. We use it as a building block of our new hybrid loss function, which addresses the limitations of the traditional NDCG measure.\n***\n\n\nQ:  Do subregions need to be given?\n\nA: A subregion is defined as the spatial neighborhood of each location including itself. Users can define subregions based on their knowledge or common assumptions. In this paper we use a common choice of 3x3 grid cells around each location. \n\n***\n\nQ: What are the quantitative measures of performance wrt baselines?\n\nA: We use four metrics to measure the performance of baselines and proposed method including NDCG, L-NDCG, top-K precision, and cross-k function:\n*  NDCG is a metric of ranking quality or the relevance of foremost important items, and a higher score indicates better ranking quality. \n*  L-NDCG is proposed in our paper and designed to measure spatially local ranking quality over every sub-region of the study area. Essentially, L-NDCG is the average of NDCG scores for all subsets of locations. \n*  Top-k precision is a widely accepted measurement in learning-to-rank problems. In our cases, it is equal to the number of locations in top-k recommendations where events occurred divided by the number of recommendations k\n*  The Cross-K function measures the spatial correlation between the predicted locations and true locations. Details can be found in Section 4.4\n\n***\n\n[1] Kingma, et al.. Adam: A Method for Stochastic Optimization. 2014\n"}]

Author rebuttal:
Q: The experimental results should be presented with error bars (RZQE)

A:  Thank you for pointing it out. We have revised the three experiment results and now report the average performance including NDCG, L-NDCG, and Precision with standard deviation over 3 runs in Table 1 and Table 2 of the updated pdf. The results show that SpatialRank significantly outperforms other compared baselines in three datasets. SpatialRank substantially outperforms SONG and ApproxNDCG and made a noticeable improvement on L-NDCG as it is considered in the objective function.

* * *

Q: Though I could follow the reasoning of computing a top-k query, the motivation why this information is enough in several applications could be motivated better in the introduction. (JmsG)

Q: The motivation for urban event ranking is weak. The paper does not highlight the significance and novelty of this problem, and why it is more important than making event predictions for each location. (yV4P)

A: Thanks for your suggestions! We will add extra explanations and citations on our motivation for formulating a ranking problem in the introduction of the revision. We believe being able to correctly rank the foremost important locations meets the real-world demand. Because deploying limited law enforcement resources to the most needed places is necessary. Chicago Police Department has utilized criminal intelligence analysis and data science techniques to help command staff determine where best to deploy resources [1]. According to Police Executive Research Forum, there were 42.7% more resignations among law enforcement but a 3.9% decrease in hiring new officers in 2021 compared to 2019 [2]. Meanwhile, the Federal Bureau of Investigation confirms that violent crime in 2020 has surged nearly 30% over 2019 [3]. Therefore, given such growth of crimes, deploying limited law enforcement resources to the most needed places is very necessary. In addition, our case study in Figure 4 of the supplementary material also demonstrates that our proposed method can prioritize the riskiest locations and capture more hotspots compared to baselines that make predictions for all the locations. This can potentially improve the deployment efficiency of police resources. 

* * *

Q:  The technical contribution is limited and the model is not novel enough. The paper should provide more details and analysis to demonstrate the advantages and challenges of the proposed approach. (yV4P)

A: In this paper, we present the following contributions in terms of problem formulation, model design, optimization strategy, and evaluation metrics.
*  This is the first paper to formulate an urban event forecasting problem as a spatial learning-to-rank problem and solve it by directly optimizing a spatial version of the NDCG measure.  (As opposed to Existing works solve this problem by optimizing non-ranking-based metrics such as cross entropy by related work).
* We propose a novel local ranking measurement named L-NDCG and integrate it into our new loss function. This is to the best of our knowledge the first NDCG-based measure that considers spatial autocorrelation of data. The new hybrid loss **is quite different from the original NDCG** in not only an additional local ranking term, but also the underlying scientific assumptions as well as the non-trivial computational techniques needed to efficiently evaluate it. In Section 3.2, we explained the first law of geography [7] that nearby locations tend to be similar, which makes it challenging to rank neighboring locations correctly. L-NDCG emphasizes ranking correctly on each subset of locations so that important locations can be distinguished from their nearby locations. This is an important advancement over existing work.
*   We propose a novel importance-based location sampling algorithm to efficiently train the model to optimize the hybrid NDCG loss function, where we guide the model to pay more attention to important locations with higher training errors.  We explain more details in Algorithm 1 in the paper. **The algorithm is also very different from traditional NDCG optimization techniques** as it uses spatial sampling to address the L-NDCG part of the loss function for the first time.
* Our proposed optimization framework can work with different deep learning architectures. We proposed a variant of the existing graph neural networks with a novel adaptive convolution layer to capture the dynamic correlations between locations. Unlike prior works such as [4], our design allows us to dynamically capture the correlations between locations. This improves the performance of the method effectively.
* We provide comprehensive experiments on our proposed approach. We evaluate the performance of the adaptive convolution layers (SpatialRank* vs SpatialRank in Table 1), the advantages of L-NDCG in the hybrid objective function (ablation study on $\sigma$ in Table 3), and the effectiveness of our spatial sampling algorithm (compared with SOTA optimization methods in Table 2). The overall results clearly show that each of the above three contributions play an important role in helping SpatialRank outperform all the SOTA baselines on three different datasets. Therefore, we believe our paper has important, novel and valid contributions.

* * *

[1] [CPD Expands Smart Policing Technology to Support Strategic Deployment and CTA Safety]( https://home.chicagopolice.org/cpd-expands-smart-policing-technology-to-support-strategic-deployment-and-cta-safety/)

[2] [PERF survey shows steady staffing decrease over the past two years]( https://www.policeforum.org/workforcemarch2022)

[3] [US murder rate continued grim climb in 2021]( https://www.foxnews.com/us/us-murder-rate-continued-grim-climb-in-2021-new-fbi-estimates-show)

[4] Zhang, Yingxue, et al. TrafficGAN: Off-Deployment Traffic Estimation with Traffic Generative Adversarial Networks. IEEE, 2019, pp. 1474–79.


Meta-review: The paper studies the important problem of ranking urban events and adopts a novel NDCG optimization on spatiotemporal data. The resulting SpatialRank algorithm shows strong experimental results. The reviewers appreciate the novelty of the method and the clarify in the presentation.
Decision: Accept (poster)