# Unsupervised Transformer-Based Anomaly Detection in ECG Signals
## 3 Materials and Methods
### Proposed Unsupervised Transformer Architecture  
In this paper, we introduce a transformer-based network for anomaly detection in ECG signals. The overall model architecture comprises two parts: an embedding layer and a standard transformer encoder (see Figure 3). All normal time series data of ECG signals, which take the form of a 2D tensor of sequence length \(\times\) number of features, were first encoded into sequences of embeddings, which were then fed into a multilayer bidirectional transformer network to produce their corresponding representations. The final linear dense layers predicted the input ECG signal with the same input sequence length. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Dataset** & **Normal** & **Anomalous** & **Total** \\ \hline Training data & 12,045 & 0 & 12,045 \\ Validation data & 3012 & 0 & 3012 \\ Test data & 3767 & 2115 & 5882 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Overview of each set number of sequences on the MIT-BIH Arrhythmia dataset. Figure 2: Raw signal data (**a**) before the median filter and (**b**) after applying the median filter. The embedding layer was introduced for higher dimensionality mapping of input tensors into a feature space. Because the input dimension was linked to the size of the transformer's hidden layer, we partitioned this layer into two parts: a linear dense layer, which projected the input into a higher-dimensional vector, and a dropout layer to avoid overfitting. Sinusoidal positional encoding was used to encode the order of the input sequences:  
\[\text{PE}_{(\text{pos,2i})}=\sin(\text{pos}/10000^{2i/d_{\text{model}}}) \tag{1}\]  
\[\text{PE}_{(\text{pos,2i+1})}=\cos(\text{pos}/10000^{2i/d_{\text{model}}}) \tag{2}\]  
where pos is the position, \(i\) is the dimension and \(\text{d}_{\text{model}}=\) the embedding size of the model. In searching for the best combination of model configurations, we tested both one and two transformer encoder blocks. The first transformer encoder block created equivalent hidden representations of embedding sequence input. Then, to iteratively produce higher-level representations, these representations were given as input to the second transformer encoder block. A single transformer block (shown in Figure 3) included two main sublayers: a position-wise fully connected feedforward network and a multihead self-attention layer (FFN). Both sublayers adopt a residual connection [35] and layer normalization [36]. As illustrated in Figure 4, each of the h parallel scaled dot-product attention layers that make up a multi-head attention layer is referred to as a head [20]. A query vector and a set of key-value pairs were transformed into an output vector using scaled dot-product attention. \[\text{Attention}(\text{Q},\text{K},\text{V})=\text{softmax}\left(\frac{\text {QK}^{\text{T}}}{\sqrt{\text{d}_{\text{k}}}}\right)\text{V} \tag{3}\]  
where Q, K and V stand for the matrices stacked by several queries, keys, and value vectors as rows, respectively, and the dimension of the query/key vectors is \(\text{d}_{\text{k}}\). Before  
Figure 3: Structure of the unsupervised transformer architecture. calculating attention, multihead attention maps Q, K and V onto various lower-dimensional feature subspaces using various linear dense layers. Using an additional dense layer, the outputs from h heads are concatenated and projected onto a final hidden representation, as shown below:  
\[\text{MultiHead}(\text{Q},\text{K},\text{V})=\text{Concat}(\text{head}_{1}, \ldots,\text{head}_{\text{h}})\text{W}^{\text{O}} \tag{4}\]  
where  
\[\text{head}_{\text{i}}=\text{Attention}\left(\text{Q}\text{W}_{\text{i}}^{ \text{Q}},\text{K}\text{W}_{\text{i}}^{\text{K}},\text{V}\text{W}_{\text{i}}^ {\text{V}}\right) \tag{5}\]  
The weight matrices of each head's inner dense layers are denoted by \(\text{W}_{\text{i}}^{\text{Q}}\), \(\text{W}_{\text{i}}^{\text{K}}\), and \(\text{W}_{\text{i}}^{\text{V}}\), while that of the top dense layer is denoted by \(\text{W}^{\text{O}}\).