# RoPT: Route-Planning Model with Transformer
## 4 Methodology
### Sequence Feature Aggregation Module  
The purpose of this module is to capture temporal latent representations \(M_{T}\in\mathbb{R}^{l\times h}\) of current route \(R\) with Transformer. Since the length of original routes \(R\) are different, the route \(R\) with length less than \(l\) should be padded to \(l\), called \(R^{*}\), as shown in Formula (5). Firstly, according to randomly initialized matrix of node features \(M_{Emb}\), the \(Lookup\) function is called to search the spatial potential representations of current route \(R\), named \(M_{R}\in\mathbb{R}^{l\times e}\), which is shown in Formula (6). With the \(Lookup\) function, the corresponding row of \(M_{Emb}\) is extracted according to the node ID, to obtain the representation matrix \(M_{R}\) of \(R\). \[R[i]=\left\{ \begin{array}{l}R^{*}[i]\text{; }if\text{ }i\text{ }\leq|R^{*}|\\ \text{pad; }if\text{ }i\text{ }>|R^{*}|\end{array} \right. \tag{5}\]  
\[M_{R}=Lookup(M_{Emb},R) \tag{6}\]  
Then, a mask vector of the route \(R_{mask}\) is introduced to manually avoid obtaining representations of nodes in the padded positions, as shown in Formula (7). Here, \(R_{mask}\in\mathbb{R}^{l}\) denotes the mask vector of \(R\), the values of the padded positions are set as -inf to facilitate normalization of the attention mechanism, and those of rest positions are set as 0. Then, in Formula (8), \(R_{mask}\) is converted into a mask matrix \(M_{mask}\) by concatenating the route mask vector \(R_{mask}\) for \(l\) times, where _rbind_ represents line-based concatenation. Next, \(M_{mask}\) and \(M_{R}\) are fed into the Transformer to obtain latent state \(M_{T}\) of the route, as shown in Formula (9). \[R_{mask}[j]=\left\{ \begin{array}{l}0\text{; }if\text{ }j\text{ }\leq\left|R^{*} \right|\\ -inf\text{; }if\text{ }j>|R^{*}|\end{array} \right. \tag{7}\]  
\[M_{mask}=rbind(\underbrace{R_{mask}^{T},R_{mask}^{T},\ldots,\text{ }R_{mask}^{T}}_{l} \tag{8}\]  
\[M_{T}=Transformer(M_{R},M_{mask}) \tag{9}\]  
\[Transformer(\cdot)=MLP(LayerNorm(A\_N+FFN)) \tag{10}\]  
\[A\_N=LayerNorm(M_{R}+MultiHead(GenIn(M_{R}),M_{mask})) \tag{11}\]  
\[FFN=Relu(A\_N\cdot W_{1}+b_{1})W_{2}+b_{2} \tag{12}\]  
In Formula (11), residual connection and Layer Normalization are used, also represented as \(A\_N\) layer, where \(LayerNorm(\cdot)\) means the layer normalization operation. In Formula (12), the Feed-Forward layer is represented, where \(W_{1}\) and \(W_{2}\) are learnable weight matrices of the two linear layers, and \(b_{1}\) and \(b_{2}\) are biased. Next, the results are passed through an \(A\_N\) layer again to obtain final outputs of the Encoder. As shown in Formula (10), an \(MLP\) is used as a Decoder, to transform the outputs of Encoder into latent representations of current route \(M_{T}\), which is also the final output of Transformer. The following is the specific calculation process of the Transformer. Firstly, the feature matrix of current route is calculated according to Formulae (13) and (14). Here, a linear layer is used to calculate the embedding of \(M_{R}\), and position encoding of each step is added to obtain sequential features. Finally, temporal-spatial embedding matrix of route features is obtained as \(input_{E}\in\mathbb{R}^{l\times h}\), where \(M_{PE}\) donates the position matrix of the route, \(pos\in[0,1-1]\) donates the sequential step, and \(i\in[0,\text{h}-1]\) is the dimension index. \[input_{E}=GenIn(M_{R})=\text{Linear}(M_{R})+M_{PE} \tag{13}\]  
\[M_{PE}[pos,i]=\left\{ \begin{array}{l}\sin\left(\frac{pos}{10,000\frac{i-1}{ k}}\right);if\text{ }i\%2=0\\ cos\left(\frac{pos}{10,000\frac{i-1}{k}}\right);if\text{ }i\%2=1\end{array} \right. \tag{14}\]  
Then, calculations of multi-head attention are performed, where matrices of Query (Q), Key (K), and Value (V) are obtained from the input sequence with different linear transformation layers, and each head has its own independent matrices of \(Q\), \(K\), and \(V\).