# Comparing End-to-End Machine Learning Methods for Spectra Classification
## 3 Data-Driven End-to-End Neural Network Methods
### Convolutional SCT Attention Network
#### 3.2.1 Spatial Attention Module and Channel Attention Module  
In the lower branch (also the main branch), two convolutional modules Conv 1 and Conv 2 are first applied to extract local features. They have the same structure as the convolution modules in the CNN model, consisting of a 1D convolutional layer, a BatchNorm1d function, a ReLU activation function, and a MaxPool1D operation. The number of filters of the convolutional layer in Conv 1 and Conv 2 is 64 and 8, respectively, the kernel size is 5 and 5, respectively, and the step size is 2 and 1, respectively. The size of the kernel and the stride of the MaxPool1D operation in these two modules are 2. In order to facilitate the follow-up Multi-Head attention operation, the embedding module is added, which contains a fully connected layer, and a 1D layer normalization operation. It produces an output of dimension \(d_{emb}=256\). We denote the input shape of this branch as of \((B,C_{in},N)\), where \(B\) is the batch size, \(C_{in}\) is the number of channels, equal to 1 in the input, and \(N=4023\) is the number of features. After embedding the module, the output is of shape \((B,C,E)\), where \(E\) is the number of embedded features, \(E=d_{emb}=256\). In this branch, two parallel attention modules are applied, a spatial attention module and a channel attention module. The spatial attention module is designed to model the spatial dependencies between any two local features. The channel attention module is designed to capture the global interdependencies over any two channel maps introduced by the Conv 1 and Conv 2 module. These two attention modules are performed concurrently and simply fused through element-wise addition. Specifically, the Multi-Head spatial attention module is designed on [26]. If the batch dimension is not considered, we can denote the input of the attention module as \(\mathbf{X}\in\mathbb{R}^{C\times E}\), then Multi-Head spatial attention can be expressed as:  
\[ \begin{split} MultiHead(Q_{S},K_{S},V_{S})&=S_{S}W _{s}{}^{O}\\ &=Concat\Big{(}head_{1}^{S},\cdots,\ head_{i}^{S},\cdots,\ head_{h}^{S}\Big{)}W_{s}{}^{O},\\ \end{split}  \tag{1}\]  
\[ \begin{split}\text{{Mhead}}_{i}^{S}&=Attention \big{(}Q_{S}W_{i}{}^{Q_{S}},K_{S}W_{i}{}^{K_{S}},V_{S}W_{i}{}^{V_{S}}\big{)} \\ &=softmax\Big{(}\frac{Q_{S}W_{i}{}^{Q_{S}}(K_{S}W_{i}{}^{K_{S}})}{ d_{h}}^{T}\Big{)}V_{S}W_{i}{}^{V_{S}}\end{split}  \tag{2}\]  
where \(W_{i}{}^{Q_{S}},W_{i}{}^{K_{S}}\in\mathbb{R}^{d_{emb}\times d_{h}}\), \(W_{i}{}^{V_{S}}\in\mathbb{R}^{d_{emb}\times d_{v}}\) and \(W_{s}{}^{O}\in\mathbb{R}^{hd_{v}\times d_{emb}}\). In this work, we employ \(h=2\) parallel attention heads, with \(d_{v}=d_{k}=128\). At last, a residual connection is added, which can be expressed as:  
\[A_{S}=\mathbf{X}+MultiHead(Q_{S},K_{S},V_{S}) \tag{3}\]  
Here, the query, key, value \(Q_{S},\ K_{S},\ V_{S}\) used to calculate Multi-Head attention are obtained directly from \(\mathbf{X}\) without transformation. The spatial attention matrix \(S_{S}=Concat\Big{(}softmax\Big{(}\frac{Q_{S}W_{i}{}^{Q_{S}}(K_{S}W_{i}{}^{K_{S }})}{d_{k}}\Big{)}\Big{)}\in\mathbb{R}^{d_{emb}\times d_{emb}}\). According to Equation (3), the obtained spatial feature \(A_{S}\) is a weighted sum of all local features and original features. It achieved global spatial interdependencies by selectively aggregating spatial contexts according to the spatial attention matrix \(S_{s}\). The channel attention module is designed based on [63], and we extend their method from three-dimensional image processing to a 1D time series analysis. The channel attention matrix is calculated through \(S_{C}=softmax\big{(}Q_{c}K_{c}T\big{)}\) which can be described as:  
\[A_{C}=\mathbf{X}+\alpha\cdot S_{C}\cdot V_{c}=\mathbf{X}+\alpha\cdot softmax \Big{(}Q_{c}K_{c}T\Big{)}V_{c}, \tag{4}\]  
where \(Q_{C},K_{C},V_{C}=\mathbf{X}^{T}\), and \(S_{C}\in\mathbb{R}^{C\times C}\), it models the dependencies or similarity between any two channels. These two attention modules are fused together through element-wise addition.