## 3 BAT-Transformer: Bus Arrival Time with Transformer Encoder
### Stage 2: Time Prediction  
The transformer-encoder model consists of several essential components designed to predict bus arrival times, each of which contributes to the characteristics of the training data and the accuracy of the prediction. As shown in Figure 6, the input is data of the form (N, 59), where N is the number of data and 59 contains 59 characteristics, such as recorded time, distance to the stop, and frequency interval. This input is transformed into a 256-dimensional vector via a linear projection layer, which allows for more complex relationships to be learned. Before entering self-attention, it goes through a learnable positional encoding that adapts during training to better capture temporal dynamics or other specific contexts, instead of a fixed positional encoding that does not take into account existing ordering information. This interaction in the model begins by creating a Query, Key, and Value vector. Each attribute of the input data forms a Query, Key, and Value vector, which allows the model to understand the data from multiple perspectives. The query and key vectors RecordedAtTime and ScheduledArrivalTime help the model understand how late the bus is currently running, DistanceFromStop and RoadType help the model understand what conditions the bus is facing on the road, and rush_hour and day_type provide values to help the model understand external conditions, such as transportation patterns over time. The interaction between these vectors is done through multi-headed attentions, where each attentional head learns the relationships between the input data from different perspectives: temporal relationships to understand when delays occur by analyzing the difference between the recorded time and the scheduled arrival time; spatial relationships to analyze how certain conditions (such as the type of road or the condition of the bus stop) affect the progress of the bus, taking into account the remaining distance; and external conditions, such as rush hour, to consider the amount of transportation at each time of day, which determines how much the bus will be delayed. The dot product between the query and key vectors is calculated to generate a score for the relationship between each input, which is scaled and then normalized using a softmax function. The resulting attentional weights indicate which inputs are more important in the current context. For example, during rush hour, transportation conditions play a more important role, and these weights might be set higher. The value vectors are then combined with these weights to create a  
Figure 6: Stage 2: time prediction. final output that represents a unified understanding of the various factors. In this way, each attentional head computes interactions in parallel to integrate information from different perspectives, and the results from each head are combined back together in the concat step in the form of (N, 256) to form a multi-attentional output. This structure allows the model to capture complex relationships between multiple factors simultaneously. The output of the self-attention mechanism passes through the ADD & NORM layer before being directly fed as input to the feedforward network (FFN). In situations where subtle changes in distance or transportation conditions can act as important signals, Residual Connection ensures that the original input is passed along with the transformed output to ensure that the original information is not lost so that these signals are not overwhelmed by more prominent features. Layer Normalization also normalizes the output of each layer to stabilize learning and improve learning speed. The FFN then performs more complex transformations on each input feature, performing non-linear transformations on each feature, allowing the model to learn more complex patterns.