# Short-Term and Long-Term Travel Time Prediction Using Transformer-Based Techniques
## 3 Proposed Methodology
### Short-Term and Long-Term Integrated Encoder-Decoder  
The Short-Term and Long-Term Integrated Encoder-Decoder (SLIED) forms an integral part of the SLIT framework, exploiting the potential of \(\hat{S}\) for a multi-step travel time prediction \(\hat{Y}\). SLIED is composed of two main components: the Short-Term and Long-Term Integrated Encoder (SLI-E) and the corresponding Decoder (SLI-D). The procedure is detailed in Algorithm 1. ```
1:Input: Periodic segments \(\hat{S}\), Target travel time \(Y\)
2:Output: Predicted travel time \(\hat{Y}\)
3:procedureSLI-E
4:\(S_{enc}^{\text{S}}\gets SDEB(\hat{S}^{S})\)
5:\(S_{enc}^{L}\gets LDEB(\hat{S}^{L})\)
6:\(S_{enc}\leftarrow\text{Concatenate}(S_{enc}^{S},S_{enc}^{L})\)
7:endprocedure
8:procedureSLI-D
9:\(DecInput\leftarrow\text{PE(Y)}\)
10:for each decoder layer do
11:\(MaskedOut\leftarrow\text{MSA}(DecInput)\)
12:\(CrossOut\leftarrow\text{CrossAtt}(MaskedOut,S_{enc})\)
13:\(DecOut\leftarrow\text{FFN}(CrossOut)\)
14:\(DecIn\leftarrow\text{AddNorm}(DecOut)\)
15:endfor
16:\(S_{dec}\gets DecInput\)
17:endprocedure
18:\(\hat{Y}\leftarrow\text{FC}(S_{dec})\)
```  
**Algorithm 1** Short-Term and Long-Term Integrated Encoder-Decoder. SLI-E is designed to capture a wide range of temporal patterns, addressing both long- and short-term dependencies within the data through distinct encoding blocks: the Long Short-Term Dependency Encoding Block (LDEB) and the Short-Term Dependency Encoding Block (SDEB), respectively. The design of LDEB and SDEB follows the encoder block of the original Transformer model [26]. LDEB processes the long short-term segments \(\hat{S}^{L}\), represented as an \(84\times 316\) tensor, which aggregates data from each day's short-term segments extracted by EDP. This data aggregation over consecutive days is beneficial for extracting and learning dependencies from observed patterns over extended periods. The self-attention mechanism is then applied to these periodic segments, weighing the significance of each input based on its relevance to the prediction. The design of LDEB aims to intuitively model potential influences from past traffic conditions on future predictions, hoping to enrich the model's forecasting capabilities based on observed outcomes. Consequently, LDEB's output, denoted as \(S_{enc}^{L}\), offers a representation of long-term temporal dependencies. Conversely, SDEB is designed for capturing the most recent traffic conditions close to the current time by processing \(\hat{S}^{S}\), which is formatted as a \(12\times 316\) tensor. This tensor contains the immediate short-term traffic data, representing the latest and most relevant traffic conditions. The SDEB algorithm analyzes data points up to the current moment, using a self-attention mechanism to learn from rapid changes occurring in short intervals. It assigns significance to each data point based on its impact on upcoming traffic conditions. The output of SDEB is denoted as \(S_{enc}^{S}\). The encoded long-term and short-term representations, \(S_{enc}^{L}\) and \(S_{enc}^{S}\), are integrated into \(S_{enc}\), a \(96\times 316\) tensor, which contains temporal dependencies important for both upcoming and extended travel time predictions. This integrated result, \(S_{enc}\), then serves as the input for the subsequent SLI-D stage. SLI-D uses cross-attention mechanisms to relate encoded features \(S_{enc}\) with target data \(Y\). This establishes connections between historical data and target queries, enhancing traffic time prediction. The non-autoregressive design of the decoder, enhanced by cross-attention, effectively mitigates error accumulation across extended prediction periods. The output of SLI-D, denoted as \(S_{dec}\), is processed through a two-layer fully connected neural network (FC). This FC maps the encoded features from \(S_{dec}\) into a 12x1 vector. Each element of this vector provides a travel time prediction for consecutive time steps within a designated hour.