## 4 Experiment and Results
### Results  
In this study, we applied linear projection, learnable location-encoding techniques, and a fully connected layer to a transformer-encoder model to predict bus arrival times, which was evaluated on various metrics and subjected to K-fold cross-validation. We also compared it to LSTM, GRU, Bi-LSTM, FCNN, and standard transformer-encoder models to demonstrate its performance. First, the final evaluation results of the BAT-Transformer were LOS: 8722, RMSE: 92, MAPE: 35%, with an RMSE value of 92 indicating an expected arrival delay of 92 s. The graph below shows the evolution of the evaluation metrics during the learning process. As shown in Figure 9, the blue line represents the training loss and the yellow line represents the validation loss, which is the difference between the predicted value and the actual value during model training. The graph shows that the loss value decreases rapidly as the number of epochs increases. During the early epochs, the loss value decreased significantly, indicating that the model learned quickly, and after about 20 epochs, the rate of decrease decreased, indicating that the model approached the optimal state. In the final epoch (50 epochs), the loss value remained almost constant, indicating that the model reached a steady state without significant  
the validation RMSE converges to a similar level to the training RMSE, indicating that the model performed consistently across different datasets. As shown in Figure 11, the MAPE value decreased sharply during the initial epoch, indicating that the model quickly reduced its prediction error as it learned. There was a temporary increase in the MAPE value during the middle epoch, which may have occurred as the model learned certain patterns, and as learning progressed, the MAPE value decreased again and stabilized in the final epoch, indicating that the model's prediction  
Figure 10: RMSE of training and validation: The blue line shows the change in training values and the yellow line shows the change in Validation values. Figure 9: Loss of training and validation: The blue line shows the change in training values and the yellow line shows the change in Validation values. performance remained consistent. The convergence of the validation MAPE to a similar level to the training MAPE confirms that the model's predictive performance is stable across a wide range of data patterns. These three graphs indicate that the model gradually improved with learning and ultimately achieved stable prediction performance, which supports that the proposed transformer model is effective in predicting bus arrival times. Figure 12 shows a graph comparing the training RMSE of the six models. The RMSE as a function of epoch during the training of the Transformer-Encoder, FCNN, BAT-Transformer, LSTM, GRU and Bi-LSTM models can be seen in Figure 12, and  
Figure 11: MAPE of training and validation: The blue line shows the change in training values and the yellow line shows the change in Validaion values. Figure 12: Comparison of RMSE: the red line represents FCNN, the blue line represents transformer-encoder, and the green line represents BAT-Transformer. it can be seen that the BAT-Transformer model converged faster and achieved a lower RMSE. The Transformer-Encoder model showed a stable decreasing trend, with an initial RMSE starting at about 476 and gradually decreasing to about 122 at the 50 th epoch. The FCNN model had a higher RMSE value in the early stages, which decreased rapidly, showing a sharp performance improvement, and then maintained a relatively moderate decrease. The BAT-Transformer model had the lowest RMSE from the start and continued to decline, showing very rapid performance improvement and stable characteristics. The LSTM model and Bi-LSTM improved at a stable and constant rate, while the GRU showed a similar pattern of performance improvement to the LSTM, but with a slightly higher RMSE than the LSTM in the last epoch.