## 2 Theoretical Background
### Transformer Encoder  
The Transformer network [26] consists of two main components: an encoder and a decoder. The encoder is responsible for extracting features and transforming them into rich contextual representations, while the decoder generates output sequences based on these representations [27]. However, since bearing RUL prediction is a regression task rather than a sequence generation task, the decoder is not required. Removing the decoder not only simplifies the model structure and reduces computational complexity but also allows the network to focus more effectively on modeling temporal feature sequences. As illustrated in Figure 2, the encoder consists of three key components: positional encoding, a multi-head attention mechanism, and a feed-forward neural network.  
Figure 1: Agglomerative method diagram, different letters in the figure denote different features, while the arrows indicate the directions of clustering.  
Although the multi-head self-attention mechanism in the encoder is capable of capturing relationships between different time steps, it cannot inherently model their sequential order. To address this limitation, positional encoding is introduced. For each input sequence sample, positional encoding is first applied to assign explicit positional information to each feature vector. The positional encoding is computed [28] as follows:  
\[PE_{(pos,2i)}=\sin(\frac{pos}{10000^{\frac{2i}{model}}}) \tag{4}\]  
\[PE_{(pos,2i+1)}=\cos(\frac{pos}{10000^{\frac{2i}{model}}}) \tag{5}\]  
where _pos_ denotes the position of a vector within the input sequence, \(i\) represents the dimension index of the input vector, _dmodel_ is the dimensionality of the model input, and _PE_ stands for the positional encoding vector.  
The multi-head attention mechanism is one of the core components of the Transformer architecture, as illustrated in Figure 3. By introducing multiple attention heads, the network is able to capture various types of dependencies within the sequence in parallel across different subspaces. This not only enhances the model's expressive power and its ability to perceive hierarchical relationships but also improves its robustness.  
Figure 3: Multi-head attention structure.  
Figure 2: Encoder structure.  
The computation for a single attention head [28] is given by the following equation:  
\[\left\{\begin{array}{c}Q=XW^{Q}\\ K=XW^{K}\\ V=XW^{V}\\ Attention(Q,K,V)=\text{software}\Big{(}\frac{QK^{T}}{\sqrt{d}}\Big{)}V\end{array}\right. \tag{6}\]  
where \(X\) is the input data, and \(W^{Q}\), \(W^{K}\), and \(W^{V}\) are the linear projection weight matrices for computing the query (\(Q\)), key (\(K\)), and value (\(V\)) matrices, respectively; \(Q\), \(K\), and \(V\) are intermediate transition variables. The term \(d\) represents the ratio of the input feature dimension to the number of heads. The function \(\text{softmax}(\bullet)\) denotes the Softmax operation, and \(Attention(Q,K,V)\) represents the result of processing \(X\) through a single attention head.  
The computation of the multi-head attention mechanism [28] is formulated as follows:  
\[MultiHead(Q,K,V)=\text{Concat}(h_{1},\dots,h_{n})W \tag{7}\]  
\[h_{i}=Attention(Q,K,V)_{i} \tag{8}\]  
where \(n\) denotes the number of attention heads, \(W\) is the weight matrix for the multi-head attention, \(h_{i}\) represents the output of the i-th attention head, \(\text{Concat}(\bullet)\) refers to the function used to concatenate the \(h_{i}\) results, \(\text{Attention}(Q,K,V)_{i}\) denotes the computation function of a single attention mechanism, and \(MultiHead(Q,K,V)\) represents the result of the multi-head attention mechanism.