# A Transformer-Based Approach for Efficient Geometric Feature Extraction from Vector Shape Data
## 2 Methods
### Construction of the PSRT Model  
The PSRT model consists of two main components: pre-training tasks and SRT, as illustrated in Figure 1. The pre-training phase processes raw vector shape data to initialize the SRT model effectively. This phase is structured around three distinct tasks:  
* **CMP**: Certain coordinates in the data are masked, and the model learns to predict missing values. This task helps the model understand the spatial dependencies and contexts within the shape data.
* **COC**: The model learns to correct deviations or offsets in the coordinate data, thereby enhancing its ability to refine and accurately represent shape features.  
* **CSR**: The model is challenged to reorder the coordinates to their correct sequences, teach them to recognize patterns, and maintain the integrity of the vector shape structure.  
These tasks allow the model to effectively extract and learn features by offering varied contexts and tasks without the need for labeled data, thereby preparing the input for the SRT model. The SRT model is an encoding network based on transformer architecture. Relying on a multi-head self-attention mechanism, SRT can efficiently derive feature representations directly from the sequence of coordinate points in the shape data. This eliminates the need for manually crafted features, such as statistical or geometric features, allowing for an end-to-end learning process. This capability means that SRT can capture intricate patterns and dependencies within shape data, providing a comprehensive re-expression of shape features.  
The SRT encoding network employed an N-layer structure with \(h\) attention heads. Each vector shape is serialized and converted into one-hot encoding. After passing through the embedding layer, it is combined with the position encoding and serves as the input to the feature encoding network. Position encoding is referenced from the original model and still uses fixed encoding, as shown in Equations (4) and (5) [48].  
\[\text{PE}(pos,2i)=sin\Bigg{(}\frac{pos}{1000\frac{2i}{c\_model}}\Bigg{)} \tag{4}\]  
\[\text{PE}(pos,2i+1)=cos\Bigg{(}\frac{pos}{1000\frac{2i}{c\_model}}\Bigg{)} \tag{5}\]  
Here, _pos_ represents the index of a specific element in the input sequence, while _d_model_ denotes the model dimension, indicating the size of the embedding vector.  
Upon traversing the attention heads and residual mechanisms, a single-layer encoding process was completed. The encoding module is iterated N times to yield a feature encoding output consisting of \(h\) heads and \(N\) layers.  
Graph neural networks are commonly employed for feature extraction of vector shape data due to the innate graph-like structure formed by the nodes' coordinates in geographic space. In this study, the transformer encoding network is chosen over graph neural networks since it encompasses the functionality of multiple graph neural networks,  
Figure 1: Architecture of the PSRT model, comprising pre-training tasks and SRT.  
enabling superior extraction of shape features without reliance on artificially designed statistical features.  
In a multi-head attention mechanism, a specific attention head \(h_{i}\) computes attention weights using Query (\(Q\)), Key (\(K\)), and Value (\(V\)) matrices. These matrices are derived from the input sequence through linear transformations [48]. The specific computation formulas are as follows:  
\[\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{Q\cdot K^{T}}{\sqrt{d_{k}}} \right)V \tag{6}\]  
By integrating these components, the PSRT model leverages the strengths of transformer architectures to achieve efficient and automated feature extraction from vector shape data, thereby setting the stage for application across various downstream tasks.