# Visual Analysis of Spatiotemporal Data Predictions with Deep Learning Models
## 2 Related Work  
Many researchers desire to understand how deep learning models are trained, how model representations are interpreted, and how deep learning supports decision making [9]. The idea of model understanding in machine learning is divided into interpretability and explainability [10]. The interpretation is to understand the status transitions that occur while changing input or algorithm parameters in machine learning models. Explainability is the interpretation of the internal mechanisms of machine learning models in understandable human terms. In visualization and visual analytics (VA) areas, some studies have been proposed to support the design and debugging of models by applying VA to an interactive machine learning workflow [9]. In the area of model interpretation, visual analytics has focused on understanding the structure of models [11], analyzing the performance of predictive models [12], identifying misclassified instances [13; 14; 15], and comparing the performance of multiple predictive models [16]. To explain the structure of the model, node-link diagrams [17], drawing directed graphs [11], and directed acyclic graphs [18] are applied. [PERSON] et al. [11] presented a TensorFlow graph visualizer to assist in understanding machine learning architectures. [PERSON] et al. [18] proposed a visual analytics system to understand and diagnose a convolutional neural network, using a directed acyclic graph. Although many visual analysis systems support machine learning modeling, most are limited in classification models. Therefore, we believe that our system assists us in understanding deep learning modeling while improving spatiotemporal predictions. The performance analysis of the predictive model includes studies to explore the combination of input features [19] and to improve the quality of the labeled data [13; 20]. [PERSON] et al. [13] introduce a system for correcting false labels in training data, using hierarchical visualization with incremental t-distributed stochastic neighbor embedding (t-SNE). If we can observe the cause and consequence of the predictive model in interactive machine learning, the explainable AI (XAI) must be able to analyze why the model makes such a decision [21]. To understand the internal mechanism, researchers detect errors or weight changes observed in specific output changes during the learning process based on the performance metrics [22]. Comprehensive theoretical studies of the role of visual analytics in deep learning have been conducted, and it is possible to interpret various deep learning models, such as CNN [23], DNN [24], RNN [25; 26], LSTM [27; 28], and DQN [29]. [PERSON] et al. [22] also presented an interactive and explainable visual analytics framework for understanding machine learning models. They can diagnose and improve the limitations of the designed model through quality monitoring, provenance tracking, and model comparison in the TensorBoard environment. In the field of statistics, time-series data predictions are mainly performed with the autoregressive model, moving average model, and autoregressive moving average (ARIMA) model. In machine learning studies, the RNN and LSTM are known to be suitable for time series prediction. LSTM models can be constructed according to the layer layout, structure, connectivity, and combination with other neural networks. Typical LSTM models are Vanilla LSTM [30], Stacked LSTM [31], Bidirectional LSTM [32], etc. Although the LSTM model generally outperforms the ARIMA model in time series prediction [33], the ARIMA model outperforms the LSTM in time series data with strong seasonal factors [34]. Studies for the interpretation of LSTMs and RNNs were published in the visual analytics community. [PERSON] et al. [35] visualized the behavior of LSTM and GRU in speech recognition and presented that LSTM has long-term memory but is more sensitive to noise than RNN. [PERSON] et al.