# Comparing End-to-End Machine Learning Methods for Spectra Classification
## 3 Data-Driven End-to-End Neural Network Methods
### Convolutional SCT Attention Network
#### 3.2.2 Temporal Attention Module  
The temporal attention module is illustrated in the upper branch of Figure 3. It is introduced to model the interconnections of any two spectral time series. Before applying the Multi-Head temporal attention, the input spectra are first fed into an FC embedding module and a position embedding module. Its dimension of output \(\mathbf{X}_{T}\) is set to \(d_{emb}^{T}=d_{emb}=256\), which is the same as the output dimension of the embedding module in the spatial and channel attention path.  
The shape of input in the upper branch is \((L,B_{t},N)\), where \(L\) is the sequence length, and \(B_{t}\) is the batch size in the temporal attention branch. Similar to the Transformer-based model, the input sequence length \(L\) is set to 31 during training and 349 during testing, which corresponds to the number of original training examples and the total number of spectral curves in the test dataset. The Multi-Head temporal attention is expressed as:  
\[MultiHead(Q_{T},K_{T},V_{T})=Concat\Big{(}head_{1}^{T},\cdots,\ head_{i}^{T}, \cdots,\ head_{l}^{T}\Big{)}W_{T}{}^{O}, \tag{5}\]  
\[\begin{split} head_{i}^{T}&=Attention\Big{(}Q_{T}W _{i}{}^{Q_{T}},K_{T}W_{i}^{K_{T}},V_{T}W_{i}^{V_{T}}\Big{)}\\ &=softmax\Bigg{(}\frac{Q_{T}W_{i}{}^{Q_{T}}\big{(}K_{T}W_{i}^{K _{T}}\big{)}^{T}}{d_{k}^{T}}\Bigg{)}V_{T}W_{i}^{V_{T}},\end{split} \tag{6}\]  
where \(W_{i}{}^{Q_{T}},W_{i}^{K_{T}}\in\mathbb{R}^{d_{emb}^{T}\times d_{k}^{T}}\), \(W_{i}{}^{V_{T}}\in\mathbb{R}^{d_{emb}^{T}\times d_{v}^{T}}\) and \(W_{s}{}^{O}\in\mathbb{R}^{d_{v}^{T}\times d_{emb}^{T}}\). In this work, we employ \(l=8\) parallel temporal attention heads. At last, a residual connection is added, which can be expressed as:  
\[A_{T}=\mathbf{X}_{T}+MultiHead(Q_{T},K_{T},V_{T}) \tag{7}\]  
The temporal query, key, value \(Q_{T}\), \(K_{T}\), \(V_{T}\) used to calculate the temporal Multi-Head attention are obtained from the \(\mathbf{X}_{T}\). The temporal attention matrix \(S_{T}=Concat\Big{(}softmax\Big{(}\frac{Q_{T}W_{i}{}^{Q_{T}}\big{(}K_{T}W_{i}^{ K_{T}}\big{)}^{T}}{d_{k}^{T}}\Big{)}\Big{)}\in\mathbb{R}^{L\times L}\). According to Equation (7), in our application, the shape of the obtained temporal feature map \(A_{T}\) is (31, 31) during training, and (349, 349) during testing.  
After calculating Multi-Head temporal attention, we reshape the output into the shape of (B, 1, E), and apply a 1 \(\times\) 1 Conv 4 module. The convolutional layer in the 1 \(\times\) 1 Conv 4 module serves as a scaling function for embedded temporal attention features, instead of changing the depth of feature maps as in the 1 \(\times\) 1 Conv 3 module.  
The temporal features with attention are also simply fused with the other two attention modules through element-wise addition. By applying the 1 \(\times\) 1 convolution modules, the features in these three attention modules can be easily fused.  
After that, a residual feed forward module with the same structure as in [26] is applied to further capture the features. Finally, an FC layer with the sigmoid activation function is applied to the classification task.  
In fact, the attention modules designed in this architecture can also be used individually or concurrently in combination with any other models applied in our work (i.e., FC, ResNet, LSTM, et al.). If we only model connections across spatial and channel dimensions, we can obtain another architecture, named the convolutional SC attention model, which discards the upper temporal attention branch, as illustrated in Appendix A.