## 3 SPT-AD: Self-Supervised Pyramidal Transformer Network-Based Anomaly Detection  
### Model Framework  
Self-Supervised Pyramidal Transformer Network-Based Anomaly Detection (SPT-AD) is a transformer-based algorithm that performs time series anomaly detection by generating anomaly data through self-supervised learning. An algorithm called Anomaly-BERT was proposed based on the motif of BERT, which uses only the encoder structure of the transformer [46]. In this paper, we propose a new architecture SPT-AD based on the AnomalyBERT architecture, which replaces the core component of the multi-head self attention (MSA) with the Pyramidal Attention Module (PAM) of Pyraformer and adds a Coarse-Scale Construction Module (CSCM) to improve the anomaly detection performance and computational efficiency of time series data [47]. The proposed model overcomes data imbalance while maintaining synthesis outliers through a data degradation method and effectively learns temporal and inter-variable correlations utilizing multi-resolution representation through the Pyramidal Attention Module. This section describes the proposed model structure and algorithms in detail. The overall structure of the model is shown in Figure 1.  
SPT-AD uses four synthesis outlier methods proposed in AnomalyBERT to generate outlier data and learn outlier patterns by comparing the original data with the degraded data when training the model. The four types of synthesis outliers used are as follows. They can also be seen in Figure 2.  
Our approach to data degradation methods include the following elements:  
* Soft Replacement: Soft replacement is a method of replacing missing values or outliers in time series data utilizing a specific combination of weights for values outside a window that contains data around those values. * Uniform Replacement: Uniform replacement is a method of replacing missing values or outliers with a single, fixed, constant value. This value can be chosen as a logical default for the data, or it can be the mean, median, or a specific reference value. Figure 1: SPT-AD framework. * Length Adjustment: Length adjustment is a technique for adjusting the length of time series data, stretching or shrinking the data to make them the right length when they do not meet your analysis needs. * Peak Noise: Peak noise is a technique for inserting unusually high peak values at specific points in time series data; it is useful for testing the noisy nature of data or observing the response of a system. In the Linear Embedding Layer Map, each point in the time series data is embedded into a high-dimensional space to create an initial embedding vector as input for the transformer model. Input data are denoted as \(X=[x_{1},x_{2},\cdots,x_{n}]\in\mathbb{R}^{N\times D_{\text{input}}}\), where \(N\) is the length of the time series, and \(D\) is the dimension of each data point. Output data are denoted as \(X_{\text{embed}}=[e_{1},e_{2},\cdots,e_{N}]\in\mathbb{R}^{N\times D_{\text{ embed}}}\). The Coarse-Scale Construction Module (CSCM) is a module that converts time series data into multi-resolution layers, compressing the information in the time series at each resolution and passing it on to higher layers. Each resolution is generated by 1D convolution using a kernel and stride of \(C\)-Scale. If the input time series data are \(X_{\text{embed}}\in\mathbb{R}^{N\times D_{\text{embed}}}\), the CSCM reduces the data length to \(N/C\) at each layer. The final output consists of a feature vector \(X_{\text{CSCM}}\) of multi-resolution layers. \[X_{\text{CSCM}}=[X_{1},X_{2},\cdots,X_{s}]\in\mathbb{R}^{(N+\frac{N}{C}+\cdots +\frac{N}{C^{3-1}})\times D_{\text{hidden}}} \tag{1}\]  
(\(S\): number of resolution layers, \(C\): convolution stride). The transformer body is composed of multiple layers of Pyramidal Attention Module (PAM) and Feed Forward (Multilayer Perceptron, MLP) blocks. The Pyramidal Attention Module is used to learn temporal patterns and interactions between variables in time series data.