# Visual Analysis of Spatiotemporal Data Predictions with Deep Learning Models
## 1 Introduction  
Spatiotemporal data contain feature information, such as temporal and spatial information, at the same time [1]. Therefore, spatiotemporal correlation patterns are often utilized together in prediction models. Spatiotemporal prediction models are applied in various fields, such as traffic, weather, social media, flights, and human migration. However, creating a prediction model is challenging because each field has a different degree and type of spatiotemporal correlation and complexity [2]. Different means of recording spatiotemporal data and different data formats make predictions more complicated. Radar echo data and air pollutant data have different recording schemes and data formats. Radar echo data are signals reflected from objects, such as raindrops. Radar echo data sets can be collected in the form of a two-dimensional image sequence in a regular grid. On the other hand, air pollutant data are recorded with air-condition information from sensors. Most air pollutant data are continuously recorded in time but have uneven spatial information, due to irregular sensor locations, which is more complicated for spatiotemporal pattern extraction. In machine learning [3], the machine is trained using data and algorithms to learn how to perform a task. Deep learning [4] is considered an evolution of machine learning, which uses a programmable neural network that empowers the machine to make decisions without guidance from humans. There are two methods in machine learning, including supervised learning and unsupervised learning. The main difference between these two is the use of labeled data sets. Supervised learning utilizes labeled input and outputdata, while unsupervised learning does not. Deep learning models can be applied for temporal pattern prediction. Typically, recurrent neural networks (RNNs) use recurrent computations to train temporal patterns from historical sequence information and produce predictions. Many studies were conducted to predict spatiotemporal data with gated recurrent unit (GRU) networks and long short term memory (LSTM) networks with RNN structures [5; 6; 7], which have a looping constraint on the hidden layer of the artificial neural network (ANN). Preprocessing is expected to handle spatiotemporal data as input to the RNN architectures. Since the RNNs do not consider the spatial structure, the spatial information within the data may be dropped during the preprocessing. A spatiotemporal predictive deep-learning model was proposed to resolve the problem in RNN that does not consider the spatial structure. The convolutional LSTM network [8] recognizes the spatiotemporal correlation by combining the LSTM layer and the convolutional layer. Although this deep learning model predicts spatiotemporal data adequately, it is puzzling to understand how the incorporation of spatial information in the input data can improve the predictive performance of the deep learning model, just by reviewing the accuracy. Since the spatial information contained in each feature of the data is different, the prediction performance also varies, according to the feature selection. In addition to the feature selection, the incorporation of spatial information, such as grid structure, also affects the deep learning performance. Therefore, it is challenging to interpret deep learning results that depend on input characteristics such as feature selection, temporal correlation, and spatial correlation. The more difficult the deep learning model is to interpret, the more time-consuming the modeling process is. Hence, it is necessary to develop a system that allows us to quickly interpret the output of the deep learning model, according to the input characteristics. The contributions of our work are as follows. * We develop a visualization system to support the interpretation of outputs from deep learning models. * We propose multiple feature selection functionalities with temporal and spatial information.