# Short-Term and Long-Term Travel Time Prediction Using Transformer-Based Techniques
## 2 Related Work
### Transformer Models in Traffic Prediction  
Transformers have broadened the scope of traffic prediction research in ITS, showing notable versatility in managing sequential data and capturing long-range dependencies [12; 13]. They are effectively applied in a range of applications from travel time prediction [14] to traffic flow [1; 4; 15] and speed analysis [16], demonstrating their capability to process diverse data types and address complex traffic system needs. Studies [12; 17] have illustrated their ability to capture complex spatiotemporal patterns and adapt to various scenarios, including those integrating weather data [1; 4]. Such adaptability not only demonstrates their potential to address multifaceted traffic system challenges but also demonstrates their ability to incorporate external factors, such as weather conditions, into their analysis. The notable effectiveness of Transformers across various settings, particularly in their ability to manage long-range dependencies, highlights a holistic approach to understanding traffic behavior.  
Transformer-based models have introduced innovative strategies that significantly improve over traditional forecasting methods, particularly by capturing intricate temporal relationships more effectively. One such advancement, the Informer model [18], tackles the inherent issues of traditional Transformers related to quadratic time complexity and substantial memory requirements. Employing a ProbSparse self-attention mechanism along side a generative decoder, the Informer model significantly boosts the speed of processing long data sequences, improving computational efficiency. Further, Spatial-Temporal Transformer Networks (STNNs) [19] leverage dynamically directed spatial dependencies and long-range temporal dependencies. This approach emphasizes the capacity of Transformer architectures to navigate the highly nonlinear and dynamic spatiotemporal dependency characteristic of traffic flows. Additionally, MultiResFormer [20] introduces a dynamic approach to modeling time series by selectively choosing the optimal lengths of data patches. This method demonstrates the flexibility and efficiency of Transformer models. Meanwhile, TS-Transformer [21] makes enhancements to overcome the common limitation of slow training and inference times in Transformer models. With mechanisms like the Sub Window Tokenizer and a Time-Series Pre-trained Encoder, TS-Fastformer accelerates the processing speed, streamlining the workflow for faster outcomes. Moreover, PDFormer [15] introduces a novel approach for traffic flow prediction, considering propagation delays within dynamic long-range scenarios. This innovative approach broadens the forecasting framework, integrating dynamic considerations to enrich traffic flow models' predictive accuracy and relevance. These contributions signal a significant enhancement in traffic forecasting methodologies. Transformer models, with their unique strengths, have shown vast potential in long-term time series analysis, adeptly managing the dynamic and complex nature of traffic data. This capability provides more precise, reliable, and comprehensive solutions to the challenges widespread in today's ITS landscape.