## 3 SPT-AD: Self-Supervised Pyramidal Transformer Network-Based Anomaly Detection
### Model Framework
The final output of the transformer body is a latent feature vector, \(H\in\mathbb{R}^{N\times D_{\text{hidden}}}\), where \(A\) is the length of the time series window. \(N\) is the length of the time series window (number of data points), and \(D_{\text{hidden}}\) is the feature dimensionality generated by the transformer body. The Pyramidal Attention Module is a multi-resolution-based attention mechanism proposed in Pyraformer, which is used as an alternative to multi-head self attention (MSA). Our approach's attention module includes the following elements:  
Figure 2: Synthesis outliers. * Inter-scale Attention: Soft replacement replaces missing values or outliers in time series data utilizing a specific combination of weights for values outside a window containing data around those values. \[H_{\text{inter}}^{(s)}=\text{Softmax}\Bigg{(}\frac{Q^{(s)}K^{(s+1)^{T}}}{\sqrt{d}} \Bigg{)}V^{(s+1)}\] (2)
* Intra-scale Attention: This learns the interactions between neighboring nodes within each resolution, exchanging information between neighboring nodes in the same resolution succession. \[H_{\text{intra}}^{(s)}=\text{Softmax}\Bigg{(}\frac{Q^{(s)}K^{(s)^{T}}}{\sqrt{d} }\Bigg{)}V^{(s)}\] (3)  
The Pyramidal Attention Module synthesizes multi-resolution information to generate a potential feature vector, \(H_{\text{PAM}}\in\mathbb{R}^{N\times D_{\text{hidden}}}\). Combining inter-scale and intra-scale attention reduces the computational complexity of the transformer from \(O(L^{2})\) to \(O(L)\) and enables efficient learning while maintaining temporal dependencies even on long time series data. The Prediction Block is the final step in APT-AD, which calculates an outlier score (0 to 1) for each time series point from the output of the transformer body. The model is trained by comparing it to the outlier bins generated by applying synthesis outliers. The potential feature vector \(H=[h_{1},h_{2},\cdots,h_{N}]\), which is the output of the transformer body, is input into the Prediction Block, which outputs the anomaly score \(A=[a_{1},a_{2},\cdots,a_{N}]\). Each feature vector \(h_{i}\) is transformed into a single value \(a_{i}\) through a linear transformation and normalization function. \[H=[h_{1},h_{2},\cdots,h_{N}]\in\mathbb{R}^{N\times D_{\text{hidden}}} \tag{4}\]  
\[z_{i}=Wh_{i}+b \tag{5}\]  
\[a_{i}=\sigma(z_{i}) \tag{6}\]  
Figure 2.0  
Here, \(W\) is the trainable weight matrix, \(b\) is the bias value, \(z_{i}\) is the anomaly score before normalization, and \(\sigma\) is the normalization function (Sigmoid), and the final output \(A\) is expressed as the anomaly score for each time series point, where the closer the value of \(a_{i}\) is to 1, the more likely the data point is to be an anomaly. As a result, the overall behavior of the Prediction Block can be expressed as follows:  
Figure 2.1  
The Prediction Block is trained with Binary Cross Entropy Loss by comparing the predicted outlier scores with the outlier data labels generated by the synthesis outliers. The loss function is defined as follows:  
Figure 2.2(\(y_{i}\in\{0,1\}\): degraded data labels: 0 (normal), 1 (anomaly); \(a_{i}\in[0,1]\): anomaly scores output from the Prediction Block).