PREDICTION OF TOURISM FLOW WITH SPARSE DATA INCORPORATING TOURIST GEOLOCATIONS
Authors: Julian Lemmel
Zahra Babaiee
Marvin Kleinlehner
Ivan Majic
Philipp Neubauer
Johannes Scholz
Radu Grosu
Sophie Neubauer
Venue: Submitted to ICLR 2023 (2023)
Keywords: GNN
RNN
Transformer
Tourism
Tourism flow prediction

Abstract: Modern tourism in the 21st century is facing numerous challenges. One of these
challenges is the rapidly growing number of tourists in space-limited regions such
as historical city centers, museums, or geographical bottlenecks like narrow val-
leys. In this context, a proper and accurate prediction of tourism volume and
tourism flow within a certain area is important and critical for visitor management
tasks such as sustainable treatment of the environment and prevention of over-
crowding. Static flow control methods like conventional low-level controllers or
limiting access to overcrowded venues could not solve the problem yet. In this
paper, we empirically evaluate the performance of state-of-the-art deep-learning
methods such as RNNs, GNNs, and Transformers as well as the classic statistical
ARIMA method. Granular limited data supplied by a tourism region is extended
by exogenous data such as geolocation trajectories of individual tourists, weather
and holidays. In the field of visitor flow prediction with sparse data, we are thereby
capable of increasing the accuracy of our predictions, incorporating modern input
feature handling as well as mapping geolocation data on top of discrete POI data.

Under review as a conference paper at ICLR 2023
PREDICTION OF TOURISM FLOW WITH SPARSE DATA INCORPORATING TOURIST GEOLOCATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Modern tourism in the 21st century is facing numerous challenges, One of these challenges is the rapidly growing number of tourists in space-limited regions such as historical city centers, museums, or geographical bottlenecks like narrow val- leys. In this context, proper and accurate prediction of tourism volume and tourism flow within a certain area is important and critical for visitor management tasks such as sustainable treatment of the environment and prevention of over- crowding. Static flow control methods like conventional low-level controllers or limiting access overerowded venues could not solve the problem yet. In this paper, we empirically evaluate the performance of state-of-the-art deep-learning methods such as RNNs, GNNs, and Transformers as well as the classic statistical ARIMA method. Granular limited data supplied by tourism region is extended by exogenous data such as geolocation trajectories individual tourists, weather and holidays. In the field of visitor flow prediction with sparse data, we are thereby capable of increasing the accuracy of our predictions, incorporating modern input feature handling well as mapping geolocation data top of discrete POI data.
INTRODUCTION
With increasing population and travel capacities (e-g. easy to international flights) cultural tourism destinations have seen a rise in visitor counts. addition, recent needs for social distancing and attendance limitations due the global COVID-19 pandemic have confronted tourism destina- tions with significant challenges in e.g. creating and establishing sustainable treatment of the both urbanised and natural environment or e-g. preventing overcrowded waiting-line -ines. The perceptions of tourists regarding health hazards, safety and unpleasant tourism experiences may be influenced by social distance and better physical separation Sigala (2020). As far as The United Nation's 2030 Agenda for Sustainable Development UNWTO (2015) is concerned, tourism has not only the po- tential to contribute to several of the Sustainable Development Goals (SDGs), but moreover an obligation. Only by establishing sustainable tourism it will be possible to create
sustainable cities and communities (Goal 11)
responsible consumption and production (Goal 12)
decent work and economic growth (Goal 8)
Therefore, future-oriented tourism regions aim to first understand and then control visitor flows in order to
preserve and protect their natural landmarks
reduce emissions and waste result of overcrowding e.g. in parks or narrow city centers
establish sustainable energy consumption within tourist attractions
create harmony between residents and tourists
and maximise tourist satisfaction, which is directly connected to the economical wealth of the specific tourism region.
Unfortunately, many real-world problems suffer from sparse data availability due to data compliance issues, lack of data collection even lack of data transfer through stakeholders. In the end there

---

Under review as a conference paper at ICLR 2023
not enough datasets properly train state-of-the art machine learning models. On the other hand there are datasets available, where ethical considerations have to be made on whether they full-fill data privacy policies well comply the rights of tourists. In our research data collected by POIs, tourist or tourist related facilities as well as anonymized digital device data. Although tourists agree e.g. sharing entry timestamps locations, the collection of this data is often side product of services such as ticket sales, travelcards or digital apps. The latter is the most controversial dataset used in this research, since it is location data collected by Mobilephone-Apps. The dataset can be compared to Mobilephone-daiie collected from mobile network operators, which displays locations of devices throughout a specific time-period. This common practice collection of data is entirely profit-oriented, since the companies collecting this data specifically aim to sell it. Apart from the fact that such data can help to improve scientific research and overcome real-life problems, it has to be discussed whether people are aware of what they are sharing by using these services, even if these datasets do not contain direct personal data.
non personal
The question on how to improve awareness of data shared by such apps services is not answered in this research. This scientific work is focusing on what possible to achieve in the given environment considering the given data and data history in regards to tourist flow prediction, since sparse data is a wide spread generic problem.
The first step in order to control tourist flows is to predict authentic movement and behavior patterns. However, since the tourist visitor flow is affected by many factors such as the weather, cultural events, holidays, and regional traffic and hotspots throughout specific day, it is very challenging task to accurately predict the future flow Liu et al. (2018). Due to the availability of large datasets and computational resources, deep neural networks became the state-of-the-an methods in the task of forecasting time-series data Pan et al. (222T) including tourism flow applications Prilistya el al. 20200. In this work, we focus on tourist flow prediction based on a local dataset from the visitors of the tourist attractions of the city of Salzburg as well as third-party geolocation data of individual tourists. After data preprocessing and dataset preparation, we attempt to compare the performance of dif- ferent deep-learning based methods for time-series prediction with ARIMA, traditional statisties based method. According to Li and Cao Li & Cao (2018), ARIMA is the most popular classical time forecasting method based on exponential smoothing and it was made popular in 1970s when was proposed by Ahmed and Cook Ahmed & Cook 1979) to be used for short-tern freeway traffic predictions.
Deep neural networks are proven to work very well on large datasets. However, their performance can degrade when trained on limited data, resulting in poor predictions on the test set. Since limited data is common problem in tourism time-series forecasting, we perform comprehensive compar- ison of the DNNs and traditional techniques on a small dataset to reveal the shortcomings and point necessary future improvements.
DL based models. Recurrent Neural Networks (RNNs) are the state-of-the-art models for learning time-series datasets. RNNs equip the neural networks with memory, making them successful at predicting the sequence-based data. The introduction of gating mechanism to RNNs lead to the great performance of LSTM Hochreiter Schmidhuber (1997) and GRU GRUChung et al. (2014).
RNNs have limitations when facing irreuularyysamplled time-series, present at many real-world forecasting problems such as tourist flow prediction. To address this limitation, phased-LSTM Neil et al.] (0016) adds time gate to the LSTM cells. GRU-D'Che et al.] (2018) incorporates time intervals by a trainable decaying mechanism order to handle time-series with missing data and long-term dependencies.
Another approach is to introduce time-continuous models with latent state defined at all times such as CT-RNN ichi Funahashi & Nakamura (193), CT-LSTM Mei & Eisner (2017) and CT-GRU Mozer ChennesCCeennaee ct al. (017). A family of continuous- time networks are NeuralODEs Chen et al. (0018) that define the hidden state of the network as a solution to an ordinary differential equation. Some limita- tions of NeuralODEs such as non-intersecting trajectories can be aleviated by using augmentations strategies leading to Augmented- -Neuralod-- (ANODEs) Dupont et al. (2019). Continuous-time models share some favourable properties: Adaptive computation as they can be implemented by numerical ODE (ordinary differential equations) solvers and training with constant memory cost by using the adjoint sensitivity method Chen et al. (2018). In addition, they can be statistically verified
2

---

Under review as a conference paper at ICLR 2023
by using GoTube Gruenbacher al. (2022) which constructs stochastic reachtubes (=the set of all reachable system states) of continuous-tim systems and is made deliberately for the verification of countinuous-depth neural networks.
Transformer- based models Vaswani et al. (2017) have achieved great success in many tasks, starting from natural language processing Devlin al. (2018) computer vision applications Dosovitskiy et al. (202)). Due to their great capability on sequence learning and representation, many recent research have explored their performance time-series forecasting tasks, especially for datasets with long sequences and high historical information [Zhou et al.] (2020); Wu el al. (2021); ) Zhou al. (2022). Multi-head self-attention mechanism is the main component of transformer models, enabling them to extract correlations in long sequences. However, self-attention is permutation- invariant, and can lead loss of temporal dependencies.
Graph Neural Networks (GNNs) are interesting new class of Deep Learning Algorithms that allow for the inputs Convolutions which can be seen as generalization of Convolutional Neural Networks to graph structured data as opposed to being arranged in a grid. An even more fascinating type of model are temporal GNNs that combine Graph Convolutions with RNNs. Such temporal GNN models are most prominent in traffic flow prediction applications. Zhu et al Yu et al Guo et al Bai et al Li et al.
structured as graphs. Most GNN models build on the notion of Graph
Traditional techniques. For time-series forecasting with traditional techniques gressive Integrated Moving Average (ARIMA) model. ARIMA has been used in recent studies as a baseline for the evaluation of novel deep-leaming based models Yao & Cao (2020); Bi et al. (2020);
the Autore-
Li Cao (2018) : Hassani et al. (2017) and is thus selected as baseline model for this paper as well.
Since limited data is common problem in tourism time-series forecasting, we summarize the spe- cific contributions of our paper follows:
We perform comprchensive comparison of the DNNs and ARIMA, traditional tech- nique, on a real-world dataset to reveal the shortcomings and point out necessary future improvements.
The real-world dataset is considered small because of limited historical entries.
Per point-of-interest (POI), we perform granular predictions on an hourly basis, which is critical for the task tourism flow control.
We further evaluate modern DI. techniques such as Transformers and GNNs.
To the best of our knowledge, we are the first apply a wide range of DL models to tourist flow prediction.
2 RELATED WORK
Due to the importance of tourist flow prediction in the ever- weer-growing tourism industry, visitor fore- casting has gained some attention over the past years Burger et al. (200T): Hassani al. (2017). Ex- isting work examines tourist demand forecasting using Recurrent Neural Networks such LSTMs Rizal & Hartati (2016). I Li & Cao (2018) or hidden Markov Models in conjunction with deep neural networks Yao & Cao 2020). Most of these studies make predictions with only a limited set of models.
Data granularity is another important aspect of tourism data. Many studies focus long-term predictions of monthly, quarterly and yearly, or in the best case daily number of visitors Bi et al.) (2000) in large regions as city country-level tourism demand Asvikarani al. (2020). However, performing granular predictions on an hourly basis and per POI is critical for the task of tourism flow control.
2. PREDICTION OF TOURISM FLOW
Due to the importance of tourist flow prediction in the ever growing tourism industry, visitor fore- casting has gained some attention over the past years Burger et al. /aassa Hassami et al, (2017). Ex-
isting work examines tourist demand forecasting using Recurrent Neural Networks such LSTMs
3

---

Under review as a conference paper at ICLR 2023
Rizal & Hartati ( Li & Cao (2018) or hidden Markov Models in conjunction with deep neural Yao & Cao 2020). Most of these studies make predictions with limited set of models.
networks
Data granularity is another important aspect of tourism data. Many studies focus long-tent teggtterm predictions of monthly, quarterly and yearly, or in the best case daily number of visitors Bi et al. (2020) in large regions city country-level tourism demand Asvikarani al. (2020). However, performing granular predictions on an hourly basis and per POI is critical for the task of tourism flow control.
3 DATA
Two different data sources were combined to enable the use of their different features in the training of the models and prediction of future visitor counts.
The first dataset we used stems from the "Salzburg Card" which was kindly provided to us by TSG Tourismus Salzburg GmbH. Upon purchase of these cards, the owner has the ability to enter 32 different tourist attractions and museums included in the portfolio of the Salzburg Card. The dataset consists of the time-stamps of entries to each POI. Additionally, we used data about weather and holidays in Austria. The Salzburg Card dataset will be published alongside this paper in normalized and hourly aggregated fashion. A more detailed description of the dataset can be found in the Appendix.
In addition we used location data gathered through mobile phone applications of visitors to the city of Salzburg by third-party service. This dataset roughly covers app. 3% of tourists and helps us in better evaluating tourists inbetween POI's. This dataset relies on location data transmitted by mobile-phone apps and contains sparse data about tourist-locations without any destinet frequency of recording. Since we are predicting tourism flow in the city of Salzburg it comes naturally to incorporate the street graph which can be obtained from OpenStreetMap (OSM) OpenStreetMap contributors (2017). By constructing bounding- -ox around the city center containing the majority of POIS we can then query the OSM graph using the osmnx python package. The resulting graph contains 2064 nodes for the crossings and 5359 edges for the streets where edge values correspond to the lengths of the street segments. order to incorporate the location tracking data we mapped to the OSM graph. Each reported location was mapped to the nearest node of the graph followed by aggregating the total number per hour.
CoVID-19 Tourism around the globe saw huge drops during the global CoVID-19 pandemic. Starting in march of 2020 Austria started take preemptive measure prevent the spread of the virus. These travel restrictions and closings of public spaces, hotels and restaurants severely reduced the number of tourists in and around the city of Salzburg. As a consequence, prediction accuracy could be diminished when using models that have been trained on pre-CoVID data.
METHODS
For this work, we built our own dataset on hourly data collected from tourist attractions and then expanded this by including geolocation data. Including many different datasources is key challenge for this real-world prediction task. Sparse geolocation data therefore fed into our GNN model as features. With this approach aim to create models that capable of casily integrating new datasources that might be available in the future. We then perform predictions with a rich set of models and do comprehensive comparison of the results. In this section we first introduce the dataset we used for the experiments. Then we go over the methods we chose to evaluate and compare their performances.
4.1 DEEP-LEARNING MODELS
We use large set of RNN variations on the tourist-flow dataset to perform a comprehensive com- parison of the sfate-of-the art models and provide insight on their performance. The set comprises vanilla-RNN RNN, LSTM, phased-LSTM, GRU-D, CT-RNN, CT-LSTM and Neural-ODi networks. Moreover, we used Transformer model, using only the encoder part with 8 heads, 64 hidden
4

---

Under review as a conference paper at ICLR 2023
units, and 3 layers, to forecast the tourist flow. Finally, applied Attention Temporal Graph Con- volutional Networks (A3T-GCN) Zhu et al. to our prediction problem in order to utilize geolocation data of individual tourists.
All of the Neural Networks were trained with Tackpropagation-Trrou Through- Marough-Time Time and the Adam opti- mizer Kingma & Ba (2014) using the parameters given in the Appendix In order to find optimal model size, loss function, and whether to use normalized visitor counts, we did a grid search con- ducting three runs per configuration and keeping the one which achieved the lowest RMSE. The loss functions listed in Table correspond to Mean-Squared--rroo Loss respectively.
and Huber-
4.2 TRADITIONAL METHOSS
In this study we use the non-seasonal variant of the ARIMA model which does not consider the seasonal patterns time-series. This model is usually denoted as ARIMA (p.d.g), where p is the number of autoregressive terms, dl is the number of non-seasonal differences, and q the number of lagged forecast errors in the prediction equation Burger et al. (2001). A generic expression for the non-scasonal ARIMA process is given as Hyndman & Khandakar (2008):
(1 B) = ((B),,
where {et} } is white noise process, B is the backshift operator, and (22) and ((2) are polynomials of order and respectively. The parameters p. d, and define how the function will fit the given time-series which directly affects the quality of future predictions. There several recommended approaches for the manual selection of parameters and they all rely on comparing the ARIMA model fit to the real values of the time-series try and minimize the deviation between them Ahmed & Cook (179); Hyndman & Khandakar 20088: ; Hyndman & Athanasopoulos (011))
For this study, we have used the cutto.arima function from the R forecast library that automatically fits the ARIMA model with different sets of parameters and returns the best combination of p. d, and q for the given time-series Hyndmman & Khandakar (2008); Hyndman & Athanasopoulos (2018). Given that each of the 32 POls in this study has different time seriese of visitor counts, we have determined different ARIMA parameters for each POI which were the best fit for its time-series. ARIMA predictions models were then built individually for each POI by fitting the ARIMA to that POI's training dataset with the best p. d, and & parameters for that POI using the Python pmdarima library. Each time after the number of visitors is predicted for the next hour in the test data, the true value (i.e., number visitors) for that hour is added to update the existing ARIMA model and make it aware of all previous true values before making the next prediction.
4.3 PREPROCESSING
We used the Salzburg card data from years 2017, 2018, and 2019 for our first set of experiments. In order to create the time-series data, we accumulated the hourly entries each location. The data then consists of the hour of the day, and the number of entries at that hour to each of the 32 POIs.
For the DL models, we added additional features the dataset: Year, Month, Day of month, Day of week, Holidays and Weather data. For the Holiday data we used the national holidays and school holidays and count the days to the next school day. For the Weather data, we use the hourly weather data with these features: Temperature, Feels Like, Wind speed, Precipitation, and Clouds as well as a Encoded single word description of the weather (e.g. "Snow").
We performed further pre- reeprocessing by normalizing all features to values between 0 and 1. To account for seasons, we performed sine-cosine transformation for the month. Intuitively, since is a circular feature we do not want to have the values for December and January to be far apart.
Finally, We split the data into sequences of length 30, and used the data from years 2017 and 2018 as the training set, and 2019 as the test set.
Graph Neural Networks For the GNN we used the OSM graphs as illustrated in Section 3 Our dataset of tourist locations was very sparse which subsequently resulted in very sparse inputs for each node. Since we are trying to predict numbers of entries at the POIs, we added them as additional nodes to the graph connecting them to up to 5 of the nearest nodes present in the graph with
5

---

Under review as a conference paper at ICLR 2023
Figure One sample series of OSM graphs the Salzburg city center obtained from preprocessing. Encircled nodes are the special POI nodes. Color coded are the normalized aggregated entry and tracking data, where most of the nodes indicate zero (pale)
distance of 80 m. Finally, the global features such as weather and holidays are added to the graph by linear mapping from features to nodes. Each node received second feature that represents influence these global features. This way we obtained series of graphs where each sample constitutes the OSM graph with the edge values corresponding to the distance and the node values corresponding to the aggregated location data and POI entries as well as linear combination global features. One sample is visualized in Figure
For training we used both the location and POI data-sources as labels by computing a weighted sum giving more importance to the POI entry data loss 0.9 (poi + 0.1 lother. For lpoi and lother chose For inference we predicted the whole graph and discarded the nodes that do not represent POIs. Our results show partial success for the GNN approach since it was able to handle the very sparse tracking data which led to overfitting in all other models.
# Cells
Time Train (min)
only visitors MAE
external features MAE
Model
# Parameters
Pred (ms)
RMSE
RMSE
ARIMA
224
69k
5.217
7.833
ANODE
64/ 21.3k
145.6
3.01
4.599
6.965
4,410
6.663
Vanilla RNN
128 43.7k
5.9
0.18
3.958
6.321
3.802
6.160
LSTM
32 11.9k
1.5
0.24
3.713
6.209
3,630
6.113
Phased LSTM
32/ 11.8k
27.0
0.46
3.825
6.359
3.651
6.120
CT-LSTM
/2/ 19.9k
18.1
0.31
3.734
6.239
3.700
6.185
CT-RNN
128 /
57.1
0.60
3.694
6.131
3.629
5.983
GRU-D
64 27.7k
16.6
0.33
3.638
6.121
3.621
6.073
Table 1: The prediction results for models when using only visitors count when using additional features from external data. Our experimental results show that the emrors were smaller for RNNs compared to ARIMA both with and without additional features were used.

---

Under review as a conference paper at ICLR 2023
5
MAIN RESULTS
5.1 FIRST SET OF EXPERIMENTS
We performed a diverse set of experiments with ARIMA and DL models to evaluate their forecasting accuracy, execution time and prediction time and compare the models. Table | shows the Mean- Absolute-Eroor (MAE) and Moot-Mean-Squared-ae to the timeframe from 2017-2019- before COVID.
(RMSE) achieved for each method applied
For the deep-leaming models model size, corresponding number of parameters and training and prediction times of the best run are listed additionally. Since using normalized visitor counts led to better results in every single model, we omitted non-normalized models from the table. All of the models except the ANODE achieved best results when trained with MAE as the loss function. For ANODE huber loss was best. The phased LSTM uses the least number of parameters for comparable results.
All of our deep-learning models were able to outperform the ARIMA method in both metries, with and without additional features. Providing the additional features to the models resulted in a slightly better performance for the DL. models. The improvements were quite marginal because we have limited training data available and thus increasing the number of features does not pay off, or even might result in over-fittin. ARIMA was not able to use the external features and fails when given only short sequences as input. The DL models can handle shorter sequences when trained on the whole training set. Hence, in order to ensure the fairness in our comparisons, we report the results with and without using the additional features of weather and holidays for DL models.
We measured training times for the DL models and prediction times for all models on a standard workstation machine setup (4 CPU cores with 3.9 GHz, 8GB DDR3 memory) equipped with a sin- gle GPU (GeForce GTX 1050 Ti, 4GB memory). As shown in Table ARIMA took 69s to perform single prediction for all POIs and the DL took for the same task fractions of milliseconds, while having used once y minutes for training. It is not possible to directly compare training and testing runtimes between them because ARIMA does not have dedicated training step where the model is built. Instead, before every new prediction at time t, ARIMA model is fitted to the known time series up until the time t-1 (i.e., training and testing are integrated for every prediction). Therefore, our integration of ARIMA favors accuracy over prediction time. ARIMA calculations are timecon- suming, since it makes predictions for each POI separately, while the DL models are trained with the visitors to all POIs in single vector and make predictions for all at the same time. Thus, ARIMA loses the implicit data about the total number of visitors in the city.
In order to visually explore the predictions made by the models, we plotted the predictions and the ground truth for a few selected time-windows (see Figure 2). We plot the predictions made by the DL models (including the external features) with the best MAE and RMSE, which were the GRU- D and CT-RNN respectively. The prediction made by the DL models with the visitors only data was only slightly worse than the others, which is why we omit these evaluations in the plots. Our plots show that although ARIMA is out-performed by the DL methods in the average error of all predictions, there cases where it actually performs better than the other models. The plot on the Top shows the forecast and real values for the tourists entered the Funicular Railway descend which the cable car ride leading up to Salzburg Castle. As visible in the plot, the DL models show better performance, especially in the valleys where the ARIMA fails to predict the downfalls accurately. Mid shows visitor predictions for Mozart's 's Birthplace Museum around the time of New Year's ss Eve. The reduced numbers of visitors on the 1st and 2nd of January is overestimated by all our models. Finally, on the bottom the predictions for the Festival Hall Festspielhaus guided tour are shown which is sparse since it takes place once a day at 2 pm. All models fail in prediction for the second and third peak at this location. However, CT-RNN shows a very good performance in predicting the first and last peak and at least shows an upward trend for the second and third peak. ARIMA can not handle this type of sparse data at all.
5.2 EXTENDED SET INLCUDING GEOLOCATION DATA
We conducted second set of experiments on the timeframe from 2019 2021 since only for this period geolocation data of individual tourists was available. Results are presented in Table which shows for each model the number of Parameters and MAE with and without additional features and
7

---

Under review as a conference paper at ICLR 2023
Funicular Railway Down Fortress Hohensa lzburg
Ground Truth
vistors
GRUD
ARIMA
2019-10-01 15:00:00
2019-10-02 13:00:00
2019 2019-10-03 -0-03 11:00:00
2019-10 10-04 09.00.00
2019 2019-10-04 10-010-04.000 19:00:00
2019 10-05 17:00:00
time
Mozart Birthplace Museum New Year
Ground Truth
CTRNN
GRUD
2019-12-28 08:00.00
2019 12-9-12-29 18:00:00
2019 2019-12-30 12:30 16:00:00
2019-12-31 14:00.00
2020 01-001-40 12:00:00
time
Festival Hall Festspielhaus Salzburg
CTRNN
visttors
GRUD
ARIMA
2019-22227 13:00:00
2029 02-28 12:00:00
2019 03-01 10:00:00
2019-03-02 08:00:00
2029 03-02 18:00:00
time
Predicted and True visitor counts for the Funicular Railway (top) and Mozart's Birthplace Museum (mid) and Festival Hall (bottom). Predictions are computed using CT-RNN (orange), GRU-D (green) and ARIMA (red).
Figure 2:
also when using features and the geolocation data. This time we included the Transformer and GNN models, but excluded ARIMA and ANODE due computation time reasons. Since the Salzburg Card dataset for this particular timeframe contains a significantly lower number of datapoints due lockdowns enforced by the government, the numbers must be compared directly to the results discussed in the last section.
Relatively speaking, the RNN models show comparable prediction accuracy although Transformers led to the best results. Transformers can handle multi-variate data well due to the multi-head self- attention mechanism which enables them to extract hidden correlations in input, and hence get better loss after using additional features. However, they require considerably more parameters comparison to the RNN models.
For GNNs we only conducted experiments with additional geolocation data since input graph at- tributes would be even sparser defeating the point of using graph based approach. The A3T-GCN algorithm scored slightly worse prediction error in comparison. However, all other methods failed to converge when trained on the sparse geolocation data which shows the usefulness of the GNN approach.
Only with GNNs we are able to utilize sparse geolocation Data into meaningful predictions. Since there is more sparse geolocation data expected to be processed within real-life-scemante scefarsce,,rios, this the only approach to fit these needs. Also we have to note that no explicit Hyperparameter tuning was conducted which means that the GNN prediction accuracy could be further approved upon.
8

---

Under review as a conference paper at ICLR 2023
MAE
Model
# Parameters
only visitors
features
additional geolocation
Vanilla RNN
11.0k
2.75
2.58
430.88
diverging
LSTM
23.6k
2.49
2,41
422.63
diverging
Phased LSTM
23.4k
2.56
2.40
426.89
diverging
CT-LSTM
39.9k
2.51
2.40
428.47
diverging
CT-RNN
7.0k
2.54
2.48
431.74
diverging
GRU-D
27.7k
2.53
2.46
431.71
diverging
Transformer
253k
2.35
2.23
482.78 diverging
A3T-GCN
39.7k
2.71
The prediction results for 2021 after training on data from 2019 & 2020. Each model (except the GNN) trained on just the visitors, visitors and external features and visitors with both features and tracking data. Reported is the best MAE the validation set for three different initializations. Using the sparse tracking data led divergence all models except GNNs.
Table 2:
CONCLUSIONS AND FUTURE WORK
We performed thorough evaluation of deep-learning (DL) versus ARIMA, a traditional method, on the task of forecasting tourist flow time-series. We found that all of the DL models were able to outperform the established ARIMA method when using only visitor counts the training data. Ex- tending the dataset with additional features of time, holidays, and weather improved the predictions of the DL models, while ARIMA is able handle additional features. While the improvements were little, the performance might still get boosted by performing feature selection for the weather data and improving the holidays features. In terms of the prediction time, DL models meaning- fully faster than ARIMA, which is an important aspect since most real-world applications require fast inference time.
We then evaluated various DL techniques including Transformer and GNN on the timeframe where COVID hit while also incorporating sparse tracking data from mobile phones. The results showed that adding external features again led to better results. Hoewever, when adding the geolocation data, all of the RNN models including the Transformer diverged. The A3T-GCN model on the other hand, was able to deal with this additional data albeit resulting in slightly higher validation emror. This work therefore serves as proof-of-we of-ook for incorporating spatial structure using GNNs that is robust overfitting even when fed sparse data and prediction accuracy should improve lot when denser geolocation data is available.
This paper opens many avenues for future research. The most straightforward next step is to try to improve the performance of the DL methods using additional training techniques such as regular- ization learning rate scheduling. Admittedly, the ARIMA method is most suitable for predicting univariate data which is why our classical methods should be extended by also considering methods for multivariate forecasting such Vector autoregression (VAR). Furthermore, there also exist line methods (c.g. Online ARIMA Liu et al.) that allow for adjusting the model for new datapoints.
In addition, we want use the knowledge gained build specialised models which outperform state-of-the art models in terms of short-term prediction with limited data. Another direction is to work on predictions for a longer time horizon and incorporate these predictions into recommender system for tourists. Therefore, we would like to go one step further into the direction of giving tourism regions the ability to control the visitor flow.
REPRODUCIBILITY STATEMENT
In order to ensure reproducibility publish the Salzburg Card dataset alongside this paper. The tracking data used for the second set experiments can not be published due to copyright. We will also hand in our sourcecode with the paper and make available on Github.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Mohammed S. Ahmed and Allen R. Cook. Analysis of freeway traffic time-series data by using box-jenkins techniques. Transportation Research Record, (722). 1979. ISSN 0361-1981. URL ISBN: 9780309029728.
https://tris trb. ott../t......//ee///1111...
Ayu Asvikarani, Widiartha, and Made Raharja. Foreign tourist arrival forecasting to bali using cascade forward hackpropagation. Jurnal Ilmiah Kursor, 10(4), 2020. ISSN 2301-6914. doi: 10.
v1177/kurson.v0014..... 252. URL httrs://kussorjouma..rrr/////// /ttp://////////////////////aa/a/
article/view/252
Lei Bai, Lina Yao, Can Li. Xianzhi Wang, and Can Wang. Adaptive Graph Convolutional Recurrent Network for Traffic Forecasting.
Jian-Wu Wu Bi, Yang Liu, and Hui Li. Daily tourism volume forecasting for tourist attractions. Annals of Tourism Research, 83(102923): 122, 2020. ISSN 0160-7383. doi: a..0066/.ammals.2220. 102923.
C. J. S. C Burger, M Dohnal, M Kathrada, and R Law. practitioners guide time-series methods for tourism demand forecasting - a case study of Durban, South Africa. Tourism Management, 22(4):403-409 August 2001. ISSN 0261-5177. doi: 10. 10.1016/50200-557700000000000000
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific Reports, 8(1), 2018 doi: 10.10088/41198-008--4777
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential 07 07366
equations. CoRR, abs/1806.07366, 2018. URL
Junyoung Chung, Çaglar Gülçehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. CoRR, abs/1412.3555, 2014. URL
http:://axxtt org/ abs 111.2.551..5
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2018. URL httts::/arrxiv.ort
2bs/1810.2 04805
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby, An image is worth 16x16 words: Transformers for image recognition at
scale, 2020. URL https: wrw.a.iivie.. org. 2010. 11929
Emilien Dupont, Araad Doucet, and Yee Whye Teh. Augmented Neural ODEs. 0rX [cs, stat), October 2019.
Sophie Gruenbacher, Mathias Lechner, Ramin Hasani, Daniela Rus, Thomas A. Henzinger, Scott A. Smolka, and Radu Grosu. Gotube: Scalable statistical verification of contimuous-depth models.
AAAI, May 2022.
Shengnan Guo, Youfang Lin, Ning Feng, Chao Song, and Huaiyu Wan. Attention Based Spatial- Temporal Graph Convolutional Networks for Traffic Flow Forecasting. 33(01):922-99)) 929. ISSN 2374-3468. doi: 10.1669/aaai.vv333
URL
php/ AAAI wtt.//att.cle//eeee/////...
Hossein Hassani, Emmanuel Sirimal Silva, Nikolaos Antonakakis, George Filis, and Rangan Gupta. Forecasting accuracy evaluation of tourist arrivals. Annals of Tourism Research, 13:112-127, March 2017. ISSN 0160-7383. doi: a0.1016).amaal.2017.....
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735-1780, 1997.
Rob J. Hyndman and George Athanasopoulos. Forecasting: Principles and Practice (2nd Ed).
OTexts, Melbourne, Australia, second edition, 2018.
10

---

Under review as a conference paper at ICLR 2023
Rob J. Hyndman and Yeasmin Khandakar. Automatic Time Series Forecasting: The forecast
Package for R. Journal of Statistical Software, 27(3):1-22, 2008. ISSN 1548-7660. doi: 11118637/js..0027.000
Ken ichi Funahashi and Yuichi Nakamura. Approximation of dynamical systems by continuous time recurrent neural networks. Neural Networks, 6(6):801-806, 1993. ISSN 0893-6080. doi: https:
URL https: www. www.weiieedddderettt.... com/
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014. URL
http:/ /tapii//arxvv..om wwtg/bsg////oooooo.e/t..ot...ebt 1412.. 6980 cite 1.w::4412..900mmmmntt Published con- ference paper at the 3rd International Conference for Learning Representations, San Diego, 2015.
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion Convolutional Recurrent Neural 01926
Network: Mata-Driven Traffic Forecasting.
YiFei Li and Han Cao. Prediction for Tourism Flow based on LSTM Neural Network. Procedia Computer Science, 129:277-283, January 2018. ISSN 1877-0509, doi: 10. (/11///.procc.21111 076.
Chenghao Liu, Steven C H Hoi, Peilin Zhao, and Jianling Sun. Online ARIMA Algorithms for Time Series Prediction. pp.
Lingbo Liu, Ruimao Zhang, Jiefeng Peng, Guanbin Li, Bowen Du, and Liang Lin. Attentive crowd flow machines. CoRR, Abs//809.00101, 2018. URL http://aax.vv.orr/abbb/////
00101
Hongyuan Mei and Jason M Eisner. The neural hawkes process: neurally self-modulating multivariate point process. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL httppppp:ppppcpppipnng/oo wwwrips....... paper/2017/EB 01777//e4558888000eeeeee66666330
pdf
Michael C. Mozer, Denis Kazakov, and Robert V. Lindsey. Discrete event, continuous time rnns. CoRR, abs/1710.044100, 2017. /tgp:
Daniel Neil, Michael Pfeiffer, and Shib-Chii Liu. Phased Istm: Accelerating recurrent network training for long event-based sequences. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29, Curran As- sociates, Inc., 2016. URL | htRKltttss://ppccediingg http://ppttcee.ccceeepp neur ips. wwwpaper.pppcppppppttp /tteeerr/211//2 2016/file/ 5bcc843dd7 33t.:320000223333333333333333333
pdf
ManntreeeMMpppoontiuuuu openst org 2017.
Planet dump retrieved from http://planet.osm.org
Qingyi Pan, Wenbo Hu, and Ning Chen. Two birds with stone: Series saliency for accurate and interpretable multivariate time series forecasting. In Zhi-Hua Zhou (ed.), Proceedings of the Thirtieth International Joins Conference on Artificial Intelligence, IJCAI-21, pp. 2884-2891
International Joint Conferences on Artificial Intelligence Organization, 8 2021. doi: 10.24963/ 102/3.70 BRtphtts:://oo org. /10 24 24963/15c2aaa 2021/397
Main Track.
Suci Karunia Prilistya, Adhistya Erna Permanasari, and Silmi Fauziati. Tourism demand time series forecasting: A systematic literature review. In 2020 12th International Conference Information Technology and Electrical Engineering (ICITEE), pp. 156-161, 2020. doi:
10. 10.11099/CCEE499999.... 201.9.9772....
Ahmad Ashril Rizal and Sri Hartati. Recurrent neural network with extended kalman filter for prediction of the number of tourist arrival in lombok. In 2016 International Conference Infor- matics and Computing (ICIC), pp. 180-185, 2016. doi: 10.1 A...A.....11....17111
Marianna Sigala. Tourism and covid-19 19: Impacts and implications for advancing and resetting
industry and research. Journal of Business Research, 117:312-321 2020. ISSN 0148-2963. doi: http:://oo.rry///looooo////esssp ttttttt www. scwencedirect. com/
sci cetiacee/sriiicll///////11122200

---

Under review as a conference paper at ICLR 2023
United Nations World Tourism Organisation UNWTO. Tourism in the 2030 Agenda. https:// www. www.uutt.ooooooo wrw.to.orr//outi -ttp:/oouurr------2----eeeeeddd 2015. [Online; accessed -0-Seetember-
2022].
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-60080 2017.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition trans- formers with auto-correlation for long-term series forecasting, 2021. URL https://arxiv.
argg/bs/220 2 13008
Yuan Yao and Yi Cao. A Neural network enhanced hidden Markov model for tourism demand forecasting. Applied Soft Computing Journal, 94(106465):1-20 September 2020. ISSN 1568- 4946. doi: 10.1016/j 2040.10666......
Bing Yu, Haoteng Yin, and Zhanxing Zhu. Temtool Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting. Proceedings of the Twenty-Seventh Interna- tional Joint Conference on Artificial Intelligence, pp. (634-3640) doi: 10.24963/jjcai 2058/005.
URL httpt::/taaii..ooo// org httpg//as//1703 04875
Haoyi Zhou, Shanghang Zhang. Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting, 2020. URL
https://arxiv http://///.xii..ooa/o/////////// wttp://////////////// 07436
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang. Liang Sun, and Rong Jin. Fedformer: Fre-
quency enhanced decomposed transformer for long-term series forecasting, 2022. URL https:
7/arxiv. http:/.rrgabbb//// 2201 12740
Jiawei Zhu, Yujiao Song, Ling Zhao, and Haifeng Li. A3T-GCN: Attention Temporal Graph Con- volutional Network for Traffic Forecasting. URL /
A APPENDIX
A.1
DATA
The dataset we used stems from the "Salzburg Card" which was kindly provided to us by TSG Tourismus Salzburg GmbH. Upon purchase of these cards, the owner has the ability to enter 32 different tourist attractions and museums included in the portfolio of the Salzburg Card. We accu- mulated the hourly amount of entries into these points of intersst(POl'ss.
Every entry from 2017 to 2021 using thses cards was recorded and summed up in this dataset. 2017 to 2019 serves baseline of "normal" tourism activity in the city of Salzburg. The data from 2020 and 2021 is of particular interest since the visitor numbers were heavily influenced by the global CoVID-19 pandemic. This affects overall visitor numbers as well distributions among the different attractions due to travel and local restrictions.
Name. Name of the POI Timestamp. ISO 8601 timestamp. Accumulated to visitor entries per hour, if no visitors were recorded for certain timeframe the entry omitted. Visitors. Amount of card users entering the venue the next hour. Not all visitors need to use this card this data represents subset of all tourists visiting these POI's. Longitude. Lonitude of the visited POI Latitude. Latitude of the visited POI
12

---

Under review as a conference paper at ICLR 2023
A.2 HYPERPARAMETERS
Hyperparameter
Value
sequence length
30
batch size epochs optimizer
16
300
adam 1c 3
Learning-rate
B1,2
(0.9, 0.999)
1e
loss function
msc, mac, huber 32, 64, 128 True, False
model size
normalized visitors
Table Hyperparameters used in RNN training.
13


Reviews:
[{"review_id": "jAEf7fxta9", "reviewer": "Reviewer_X9rk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper uses discrete POI time series data to predict tourist flow. In this paper, the authors perform a comprehensive comparison between traditional methods such as ARIMA and deep learning methods such as RNNs and CNNs. As the first to apply deep learning methods to the tourist flow prediction problem, the authors point out the flaws and future research directions.\n", "summary_of_the_review": "In summary, this paper should not be accepted. The authors simply carry out experiments that have been on the existing traditional methods and deep learning methods instead of giving their own solution to this problem. Besides, the author's analysis in the experimental part is also insufficient. Therefore, my overall rating is reject.\n", "strength_and_weaknesses": "Strengths:\n\n1. The description of the problem is clear.\n\n2. This paper is the first to apply deep learning methods to the tourist flow prediction problem.\n\nWeaknesses:\n\n1. The innovation of this paper is insufficient. Despite experiments that have been carried out on several methods, the authors do not propose their own methods.\n\n2. The authors do not give a detailed explanation of why GNN methods seem to outperform other methods.\n\n3. The experiment part in this paper is not enough. It lacks an ablation study part and a case study part.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cl4VkCdoFM2", "reviewer": "Reviewer_xjA7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper evaluates the performance of state-of-the-art deep-learning methods such as RNNs, GNNs, and Transformers as well as the classic statistical ARIMA method to predict the tourism volume and tourism flow. Granular limited data supplied by a tourism region is extended by exogenous data such as geolocation trajectories of individual tourists, weather and holidays. ", "summary_of_the_review": "The article has introduced the idea of predicting the volume and flow of tourists by using discrete POI data, extra external factors, and geolocation data together with state-of-the-art deep learning techniques like GNNs, RNNs, and Transformers as well as traditional statistical ARIMA approaches.  ", "strength_and_weaknesses": "Strength:  \n\nThe prediction performance is compared on several deep learning methods on continuous sequential predictions.   \n\nIncluding external features to the times series dataset could slightly improve the DL models’ performance. \n\n \n\nWeakness:   \n\nThe authors mentioned “limited data is a common problem”; but no solution was discussed to overcome this challenge. Instead, Only supervised learning models were applied for training and prediction. \n\nWhen comparing the model performance on time-series prediction problem, it would be great to further compare with the naïve prediction by simply setting all forecasts to be the value of the last observations, i.e., X_t = X_{t-1}.  \n\nIn general, this work shows a comprehensive comparison across different SOTA models; however, the contribution on creating new models/approaches and theoretical insight is still missing. \n\nSome contents in the introduction and related work look redundant. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RuU9PosOe4J", "reviewer": "Reviewer_mG9t", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a comparison of different deep learning methods for tourist flow 1-hour forecast in 30 touristic locations of Salzburg city. The data is an hourly entry number of visitors at each of the 30 sites during 3 years, with addition of some information on the day (holidays or not, etc.), and then a test using sparse tourist geo localizations is also presented at the end (but the results are worse). The deep learning methods for forecasting the time series (RNN, LSTM, GRU) are also compared with a 'standard' statistic method, called 'ARIMA'. The results are showing that deep learning models are indeed capable of better forecasting than ARIMA, but the addition of golocalization data was not useful.\n", "summary_of_the_review": "While the application is of interest, the paper is not suited for publication at ICLR as no methodological novelty is presented, moreover, the application is rather simple and the paper needs important improvement and proof-reading.", "strength_and_weaknesses": "Strengths:\n- the problem is interesting and worth investigating\n- the data is released and could be useful for the community\n- several methods were tested\n\nWeaknesses:\n- the paper does not propose a methodological novelty\n- the problem is in the end very simple (single time series forecasting one step ahead), if we do not consider the geolocalization data with graphs (which currently does not work).\n- use of MAE as the loss: this is unusual as MAE is not differentiable, so usually we use MSE.. why and how did you use MAE?\n- the final goal is not clear, as the long introduction mixes Covid problems, increase of tourists problems (which are both unrelated), ethical problems due to goelocalizations bought to have more data. \n- the paper does not seem to be novel with respect to state-of-the-art in this specific field, and experiments are not clear. Please state which are methods taken from previous works, and what is your method that would be novel. Is the statistical baseline, ARIMA, used in practice in this application? What is the gold standard today in tourist forecast?\n- the forecast is only one step ahead, i.e. 1 hour, and it would be more interesting to augment this. Also, please indicate more clearly (i.e. from the start) what is the target (prediction of next hour entries).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BpxXKO8osk-", "reviewer": "Reviewer_XQLY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the problem of predicting the number of tourists visiting different attractions on an hourly basis. Using data from 32 tourist attractions in Salzburg over a four-year time span, the paper compares 7 different deep learning approaches against a classical approach (ARIMA). The paper also explores the use of additional data (sparse geo-location information from tourists and metadata about weather and holidays). ", "summary_of_the_review": "This paper is tackling an interesting problem and producing an interesting dataset. With some changes, I think this paper could be great. My primary concerns relate to experimental methodology: on the basis of the paper, can we say anything confidently about which method works best? My claim is that currently the answer is \"no\" due to the problems listed under \"weaknesses\" above.  ", "strength_and_weaknesses": "# Strengths\n* The paper addresses an interesting spatiotemporal prediction problem.\n* The paper attempts the important work of benchmarking a diverse collection of methods on a difficult task.\n* The dataset which this paper will publicly release will no doubt be useful to the community, especially since it has data for before and during COVID. \n* Investigating the use of side information (sparse geo-location information from tourists and metadata about weather and holidays) is worthwhile and interesting. \n\n\n# Weaknesses\n* The paper trains 7 different deep learning approaches with the same learning rate and reports the results. I believe it is uncontroversial to claim that tuning the learning rate is perhaps the most important step in training a deep learning algorithm. Given this, it is difficult to trust the results in the paper. \n* It is not clear whether a validation set was used or (if a validation set was used) how it was constructed. Also, which split is used for model selection? \n* The ARIMA model gets to use the true values for past time points before making a prediction. Do the deep learning models get a similar advantage? If so, how is it implemented?\n* The results are not contextualized. At what point is an improvement meaningful? Are MAE and RMSE effectively capturing what is important for the end-user? For instance, in Table 1 we see that GRU-D is better than CT-RNN by 0.01 RMSE when doing \"only visitors\" prediction. Is this a meaningful difference for the use case? Does this represent a stable difference between the models, or would the results reverse if we trained again?\n* There is no discussion of the potential for bias in the data. Is there a good reason to believe that Salzburg Card holders are a representative sample of the population of tourists? How is the mobile phone data sampled from the population? If these datasets are biased, what effects could they have on the models? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject