# Research on Safety Prediction of Sector Traffic Operation Based on a Long Short Term Memory Model
## 4 Case Study
### Time Series Forecasting without Traffic Parameters  
As introduced in Section 3, we establish three prediction models based on different methods with the purpose of comparison. Here, the number of conflict events at the last moment is taken as the only input variable in these prediction models. The experimental steps are as follows:  
(1) The division of training set and test set. For flight conflict time series, it does hold a time order dependence between observations. The common methods of cross validation would cause the crossover of temporal features. In order to test model performance, rolling origin is used, which is an evaluation technique according to which the forecasting origin is updated successively and the forecasts are produced from each origin [36]. The original data will be divided into the training set and test set in 4:3, 5:2, and 6:1, and we name it as case 1, case 2, and case 3 respectively, which is a rolling origin with a non-constant holdout sample size. (2) The training and verification process. The training set is used as input variable and the test set is used to verify model output. The grid search method is used to optimize parameters among all models. Taking the LSTM model as an example, the key parameters in model includes the number of neurons, batch size, and learning rate. The optimization process of parameters is shown in Table 2.  
Figure 7: Statistical distribution of sector traffic parameters. Due to the randomness of thresholds and weights in the neural network, different prediction methods were tested 10 times under different data division methods, and the prediction performance of each model is shown in Figure 8.  
The average error of model prediction is reported in Table 3. \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Batch Size**} & \multicolumn{2}{c}{**Neurons = 10**} & \multicolumn{4}{c}{**Neurons = 20**} & \multicolumn{4}{c}{**Neurons = 30**} \\  & **Learning Rate** & **MAE** & **RMSE** & **Learning Rate** & **MAE** & **RMSE** & **Learning Rate** & **MAE** & **RMSE** \\ \hline \multirow{3}{*}{120} & 0.1 & 0.6698 & 1.4344 & 0.1 & 0.7808 & 1.6602 & 0.1 & 1.1558 & 2.2456 \\  & 0.01 & 0.6122 & 1.4699 & 0.01 & 0.5919 & 1.3938 & 0.01 & 0.5756 & 1.3461 \\  & 0.001 & 0.5561 & 1.4566 & 0.001 & 0.6583 & 1.5871 & 0.001 & 0.5559 & 1.4510 \\ \hline \multirow{3}{*}{180} & 0.1 & 0.7416 & 1.6152 & 0.1 & 1.2039 & 2.3464 & 0.1 & 1.8482 & 3.3518 \\  & 0.01 & 0.5524 & 1.3160 & 0.01 & 0.5756 & 1.3461 & 0.01 & 0.5756 & 1.3461 \\  & 0.001 & 0.4520 & 1.0235 & 0.001 & 0.6903 & 1.7057 & 0.001 & 0.4783 & 1.0996 \\ \hline \multirow{3}{*}{240} & 0.1 & 0.6405 & 1.5370 & 0.1 & 1.7343 & 3.1770 & 0.1 & 1.8482 & 3.3518 \\  & 0.01 & 0.6475 & 1.5399 & 0.01 & 0.5929 & 1.3945 & 0.01 & 0.5756 & 1.3461 \\  & 0.001 & 0.4380 & 0.9157 & 0.001 & 0.6572 & 1.5843 & 0.001 & 0.4147 & 0.9183 \\ \hline \multirow{3}{*}{300} & 0.1 & 0.6393 & 1.5341 & 0.1 & 1.8482 & 3.3518 & 0.1 & 1.8482 & 3.3518 \\  & 0.01 & 0.6666 & 1.5906 & 0.01 & 0.6655 & 1.5899 & 0.01 & 0.6655 & 1.5899 \\  & 0.001 & 0.4130 & 0.8311 & 0.001 & 0.5005 & 1.2870 & 0.001 & 0.3901 & 0.8037 \\ \hline \multirow{3}{*}{360} & 0.1 & 0.6851 & 1.6164 & 0.1 & 1.8482 & 3.3518 & 0.1 & 1.8482 & 3.3518 \\  & 0.01 & 0.6460 & 1.5276 & 0.01 & 0.6564 & 1.5527 & 0.01 & 0.6655 & 1.5899 \\  & 0.001 & 0.4492 & 0.8721 & 0.001 & 0.4521 & 1.0801 & 0.001 & 0.4102 & 0.8308 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The optimization process of parameters. Figure 8: Prediction performance of models. As the result of case 1 shows, the prediction errors of three models are pretty large, and the performance of SVR model is slightly better than the other two models. As the proportion of training data increases in case 2, the performance of these prediction models has been greatly improved, and the average decrease in mean square error is 68.39 %. Moreover, the model performances start showing significant differences.