# Logistics Transportation Vehicle Supply Forecasting Based on Improved Informer Modeling
## 3 Problem Description and Model Design
### Principles of Informer Model Timing Prediction Algorithm  
Existing forecasting methods in the field of time series prediction mostly focus on short-term issues. However, in practical applications, longer sequences pose higher demands on the predictive capabilities of models. This paper's logistics transportation supply prediction has a length of 64, belonging to a long-sequence time series. Recent research [42] has shown that transformer models with parallel computation and attention mechanisms have great potential for improving temporal prediction accuracy but have some limitations. The Informer model is an advanced method based on the transformer architecture specifically designed for long-sequence time series prediction. It addresses the limitations of the transformer models such as quadratic time complexity, high memory consumption, and fixed encoder-decoder architecture. It primarily offers the following three significant advantages:  
* Multi-Head Sparse Attention Mechanism  
The Informer model consists of an encoder and a decoder. In the encoder, a multi-head sparse attention mechanism (prob-sparse self-attention) is proposed to replace the traditional multi-head self-attention mechanism, allowing parallel independent training of variable feature information in different spaces. Through the integration computation mechanism, overfitting is prevented to a certain extent. The self-attention mechanism, based on tuple input, performs scaled dot-product calculations using Query-Key-Value as shown below [43]. \[Attention(Q,K,V)=\textit{Soft max}(\frac{QK^{T}}{\sqrt{d}})V \tag{2}\]  
where \(Q\) represents the query vector Query, \(K\) denotes the key vector Key, \(V\) denotes the value vector Value, \(d\) represents the input variable dimension, and _Soft max_ indicates the activation function. Due to the typical long-tailed distribution of the attention mechanism arithmetic process, only a few parts of the dot product contribute a large weight to the main attention, and the influence of most parts is very small; the sparse attention mechanism can reduce the spatiotemporal complexity from \(Q\big{(}L^{2}\big{)}\) without any significant loss of precision. To evaluate the sparsity of Query, [PERSON] [43; 44] et al. used Kullback-Leibler (KL) divergence to measure the distribution distance; the larger the value of the scatter, the larger the weight of its influence, and according to the weight size can be sorted. The sparsity measure formula of the second query vector Query is shown below. \[M(Q_{i},K)=\ln\sum_{j=1}^{L_{K}}e\frac{q_{i}K_{j}^{T}}{\sqrt{d}}-\frac{1}{L_{ K}}\sum_{j=1}^{L_{K}}\frac{q_{i}K_{j}^{T}}{\sqrt{d}} \tag{3}\]To reduce complexity, an approximate formula using sparsity is employed [45], where most query vectors are replaced with the maximum value and the remaining query vectors are replaced with a uniform distribution, as shown below. \[\overline{M}(Q_{i},K)=MAX(\frac{q_{i}k_{j}^{T}}{\sqrt{d}}-\frac{1}{L_{K}}\sum_{j= 1}^{L_{K}}\frac{qik_{j}^{T}}{\sqrt{d}}) \tag{4}\]  
The computational formula for the multi-head sparse attention mechanism at this point is as follows:  
\[A(Q,K,V)=\textit{Soft max}(\frac{\overline{Q}K^{T}}{\sqrt{d}})V \tag{5}\]  
where \(\sqrt{d}\) represents the normalization factor and \(\overline{Q}\) represents the query vector with the largest complexity in sparse evaluation constituting a sparse matrix of the same size as \(Q\). * Self-Attentive Distillation Mechanism  
The Informer model's encoder is for capturing long-range dependencies between long series of time series inputs, but the feature mapping at the coding layer generates redundancy in the input values. To reduce the high spatial complexity and memory consumption caused by the long inputs, the encoder utilizes distillation to privilege the dominant high-level features. It assigns higher weights to the more dominant features, and the length of inputs is halved layer by layer. The length of the sequence is halved layer \(j\) by layer \(j+1\).