# RoPT: Route-Planning Model with Transformer
## 2 Related Work
use LSTMs to extract temporal features from the input historical trajectories; in [17], CNNs and LSTMs are connected to construct a spatio-temporal recurrent convolutional network; and, in [18], bidirectional and unidirectional LSTMs are stacked to discover the forward and backward dependencies hidden in traffic information. There are also many works focusing on the combination of GCN and RNN models to extract spatio-temporal traffic features. In [19], outputs of multiple GCNs are fed into an LSTM to predict traffic speed. In [20],to predict traffic demands, the output of GCNs and a variational auto-encoder model are concatenated as the input of Seq2 seqGRU. (3) Extraction of Transformer-based spatio-temporal features: In recent years, due to the great success of the Transformer [21] in sequence prediction, many researchers use it to replace RNN and LSTM models. In [22], [PERSON] et al. propose an encoder-decoder structure, composed of a global encoder and a local decoder, to extract and fuse global and local spatial patterns to predict traffic flow. In [23], [PERSON] et al. extract spatio-temporal features, by stacking spatial Transformers and temporal Transformers, to predict traffic flow in multiple steps. (4) Extraction of spatio-temporal features with Transformer and Graph Neural Network (GNN): In GETNext [3], the GCN is used to extract POI embeddings on a global trajectory flow graph, and the Transformer is used to recommend a next POI with combinations of the output of the GCN and other global features. In [24; 25], [PERSON] and [PERSON] et al. focus more on learning representations of nodes in the graph. For example, in GRAPH-BERT [24], a method of sub-graph sampling is proposed, where only the original features of nodes, such as sub-graph structures, and the number of hops between nodes, are considered. In SAN [25], node features are combined with position encoding learned with graph convolutions, to feed into the Transformer. In [4; 5], [PERSON], [PERSON] et al. focus on optimizing the side effect of the Transformer. For example, in GROVER [4], where the GNN is connected with the Transformer, Dynamic Message Passing Network (dyMPN) is used to enhance the generalization ability of the model. In GraphTrans [5], the Transformer is added on top of the standard GNN layer, where a summary tag ([CLS] token) is added into the input to summarize the pairwise interactions between nodes, in order to obtain global representations of nodes. In [6; 7; 26; 27; 28; 29], the GNN and Transformer are adjusted together. For example, in [26], the Laplacian eigenvectors obtained with graph convolution is used as the position encoding in the Transformer to capture the representations of edges. In GraphiT [27], a graph convolutional kernel network (G-CKN) is used to encode the local sub-graph structures as the input of the Transformer, where the relative position encoding of the kernel function in the graph can affect the attention score in the Transformer. In Graph-GPS [28], representations of a node are captured with graph operation, and fed into the parallel transformers and message-passing graph neural network (MPNN). Recently, in Graphormer [6], GEIT [7], and STGSTN [29], graph topology is fed into the Transformer with different kinds of encoding methods, such as centrality encoding, spatial encoding, and edge encoding. Although GEIT [7] was specifically designed for traffic route planning, its Divide-and-Conquer framework demonstrates limited scalability for metropolitan-scale long-distance routing scenarios. In summary, the extraction of spatio-temporal features with the Transformer and GNN is a hot-spot in traffic prediction. In this paper, we also apply it to route planning.