# TSViT: A Time Series Vision Transformer for Fault Diagnosis of Rotating Machinery
## 2 Time Series Vision Transformer
### Transformer Encoder Layer
#### 2.2.1 Muti-Head Self-Attention  
The multi-head self-attention mechanism contributes to establishing long-term dependencies between input sequences by concatenating and fusing the outputs of multiple independent single-head self-attentions using learnable parameters. A single self-attention is a mechanism used in deep learning, which allows a model to weigh different parts of the input sequence when making predictions. The multi-head self-attention enables the model to simultaneously focus on different aspects of the input, thereby enhancing its learning expressiveness and generalization ability.  
Self-attention defines three learnable weight matrices: \(W^{Q}\in R^{m\times d_{k}}\), \(W^{K}\in R^{m\times d_{k}}\), and \(W^{V}\in R^{m\times d_{v}}\). Let \(Y\in R^{n\times m}\) represent a sequence containing \(n\) embedded patches, where \(m\) is the dimensionality of each embedded patch. \(Y\) multiplies \(W^{Q}\), \(W^{K}\), and \(W^{V}\) to obtain the query matrix \(Q\), key matrix \(K\), and value matrix \(V\), respectively, as shown in Equation (6) [15].  
\[Q=YW^{Q},\ K=YW^{K},\ V=YW^{V} \tag{6}\]  
The output of self-attention is the weighted sum of \(V\), and the corresponding weight matrix can be calculated in various ways. Among them, the scaled dot-product attention is simple and easy to parallelize and does not introduce additional parameters into the model. It has been widely used, and its specific calculation is shown in Equation (7) [15].  
\[Attention(Q,K,V)=softmax\bigg{(}\frac{QK^{T}}{\sqrt{d_{k}}}\bigg{)}V \tag{7}\]  
where \(QK^{T}\) is the attention score, \(d_{k}\) is the dimensionality of \(Q\) and \(K\), and \(\sqrt{d_{k}}\) is a scaling factor. The scaling factor can avoid gradient instability when \(d_{k}\) is large.  
The single-head attention mechanism is limited by the feature space, and its modeling ability is difficult to satisfy the various complex relationships that may exist between data. Hence, multi-headed attention is needed. There are multiple independent self-attention heads in the multi-head self-attention mechanism. Each head has its own learnable weight matrix \(W^{Q}_{i}\in R^{m\times d_{kj}}\), \(W^{K}_{i}\in R^{m\times d_{kj}}\), \(W^{V}_{i}\in R^{m\times d_{vj}}\). The calculation process is shown in Equations (8)-(10) [15].  
\[Q_{i}=YW^{Q}_{i},\ K_{i}=YW^{K}_{i},\ V_{i}=YW^{V}_{i},\ i=(1,2,\ldots,h) \tag{8}\]  
\[Head_{i}=Attention(Q_{i},K_{i},V_{i}),\ i=(1,2,\ldots,h) \tag{9}\]  
\[MultiHead(Q,K,V)=Concat(Head_{1},Head_{2},\ldots,Head_{h})W^{O} \tag{10}\]  
where \(W^{O}\in R^{hd_{vj}\times m}\), \(h\) is the number of heads, and \(d_{vj}\) is the dimensionality of \(V_{i}\).  
\(Q_{i}\), \(K_{i}\), \(V_{i}\) can be regarded as the split of \(Q\), \(K\), \(V\) in single-head self-attention under different feature subspaces. The multi-head self-attention mechanism extracts the correlation between features from multiple angles and merges the information extracted by each self-attention head to obtain richer and more comprehensive feature information.  
A dropout, an RC, and an LN are added after MSA, as shown in Figure 1. The output after these operations in the \(l\)th block can be described as Equation (11).  
\[z_{l}^{MLR}=LN(dropout(MSA(z_{l-1}))+z_{l-1}),\;l=1,2,\ldots,B \tag{11}\]  
where \(B\) is the number of the basic blocks in the transformer encoder and \(z_{l-1}\) is the output of the (\(l-1\))th block.